Hi-Singers: A Comprehensive High-Quality Dataset for Expressive Audio-Driven Singing Head Synthesis

arXiv:2609.22264v1 Announce Type: cross Abstract: State-of-the-art models for audio-driven digital human generation have achieved photo-realistic results in talking-head synthesis. However, extending these models to singing-head synthesis remains challenging due to a significant Domain Gap: singing requires more exaggerated expressions, vivid jaw openings, and precise rhythmic synchronization. Current models, primarily trained on speech datasets, often struggle with "rhythmic drift" and constrained dynamics. To address this, we introduce Hi-Singers, the first large-scale, high-quality, in-the-wild video dataset specifically tailored for singing head synthesis. Hi-Singers undergoes a rigorous automated and manual filtering pipeline, ensuring strict thematic adherence, high-resolution rendering, and stable motion, resulting in 29,608 video segments totaling approximately 170 hours. We further establish a dedicated evaluation benchmark balanced across linguistic and musical styles. Extensive experiments across diverse architectures, including 3D-coefficient and diffusion-based models, demonstrate that Hi-Singers consistently and significantly improves performance across all dimensions. Specifically, it enables models to achieve superior visual realism, enhanced lip-sync consistency, and more precise rhythmic dynamics, effectively bridging the domain gap and setting a new performance standard for the singing synthesis task. The dataset is available at https://huggingface.co/datasets/CharlesZhang-USTC/Hi-Singers

Abstract:State-of-the-art models for audio-driven digital human generation have achieved photo-realistic results in talking-head synthesis. However, extending these models to singing-head synthesis remains challenging due to a significant Domain Gap: singing requires more exaggerated expressions, vivid jaw openings, and precise rhythmic synchronization. Current models, primarily trained on speech datasets, often struggle with "rhythmic drift" and constrained dynamics. To address this, we introduce Hi-Singers, the first large-scale, high-quality, in-the-wild video dataset specifically tailored for singing head synthesis. Hi-Singers undergoes a rigorous automated and manual filtering pipeline, ensuring strict thematic adherence, high-resolution rendering, and stable motion, resulting in 29,608 video segments totaling approximately 170 hours. We further establish a dedicated evaluation benchmark balanced across linguistic and musical styles. Extensive experiments across diverse architectures, including 3D-coefficient and diffusion-based models, demonstrate that Hi-Singers consistently and significantly improves performance across all dimensions. Specifically, it enables models to achieve superior visual realism, enhanced lip-sync consistency, and more precise rhythmic dynamics, effectively bridging the domain gap and setting a new performance standard for the singing synthesis task. The dataset is available at this https URL

7 pages, 6 figures, 4 tables. Accepted to the 34th ACM International Conference on Multimedia (MM '26). Dataset: this https URL

Multimedia (cs.MM); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD); Audio and Speech Processing (eess.AS)

I.4.9; I.2.10; H.5.1

arXiv:2609.22264 [cs.MM]

(or arXiv:2609.22264v1 [cs.MM] for this version)

https://doi.org/10.48550/arXiv.2609.22264 arXiv-issued DOI via DataCite (pending registration)

analiza aiqbench

Hi-Singers: A Comprehensive High-Quality Dataset for Expressive Audio-Driven Singing Head Synthesis

Subiecte conexe