analiza aiqbench
SyncVoice advances automatic video dubbing by integrating a lightweight text-visual fusion module into pretrained TTS systems to achieve superior audio-visual synchronization and zero-shot multilingual performance. This approach resolves common limitations in speech naturalness and scalability without requiring complex architectural overhauls, offering a practical path forward for high-fidelity cross-language content generation.
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
arXiv:2512.05126v2 Announce Type: replace-cross Abstract: Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
Abstract:Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Sound (cs.SD)
arXiv:2512.05126 [eess.AS]
(or arXiv:2512.05126v2 [eess.AS] for this version)
Text integral sub licența deschisă a publicației · 145 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 18 sept. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
The Machines that Make the Machines
NVIDIA · 7 oct. 2026
NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model
NVIDIA · 28 apr. 2026