analiza aiqbench
Talker-T2AV introduces a novel autoregressive diffusion framework that decouples high-level semantic reasoning from low-level modality-specific rendering to improve talking head synthesis efficiency and quality. By using a shared backbone for cross-modal logic while employing separate decoders for audio and video details, the model achieves superior lip-sync accuracy without the entanglement issues of fully coupled approaches. This architectural shift addresses key limitations in current unified generation models where acoustic signals and visual textures are unnecessarily fused during denoising.
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
arXiv:2604.23586v2 Announce Type: replace-cross Abstract: Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
Abstract:Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)
arXiv:2604.23586 [cs.CV]
(or arXiv:2604.23586v2 [cs.CV] for this version)
Text integral sub licența deschisă a publicației · 182 cuvinte
Subiecte conexe
AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation
arXiv · 9 sept. 2026
Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
arXiv · 7 oct. 2026
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · 13 oct. 2026
[AINews] Meta Connect 2026: Muse glasses, voice, video, and Charm
Latent Space · 24 sept. 2026