aiqbench analysis
This research introduces DLLM-VSR, the first framework to apply Diffusion Large Language Models to Visual Speech Recognition by replacing traditional autoregressive decoding with iterative masked denoising. By decoupling visual-to-text alignment from length modeling and employing confidence-based unmasking, the method achieves a state-of-the-art 19.4% word error rate on the LRS3 benchmark using only that dataset for training. The approach effectively mitigates premature token decisions in ambiguous visual contexts, offering a significant architectural shift for multimodal speech processing.
Diffusion Large Language Models for Visual Speech Recognition
arXiv:2605.28456v2 Announce Type: replace Abstract: Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap compared with an upper-bound setting where the ground-truth transcript length is provided at inference, allowing the model to focus on transcript content decoding. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses and reranks the decoded candidates using length plausibility and decoding confidence. The proposed method achieves a 19.4% word error rate on LRS3, establishing state-of-the-art performance among methods using only LRS3 as labeled training data.
Abstract:Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap compared with an upper-bound setting where the ground-truth transcript length is provided at inference, allowing the model to focus on transcript content decoding. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses and reranks the decoded candidates using length plausibility and decoding confidence. The proposed method achieves a 19.4% word error rate on LRS3, establishing state-of-the-art performance among methods using only LRS3 as labeled training data.
Accepted to EMNLP 2026. Code: this https URL
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Audio and Speech Processing (eess.AS)
arXiv:2605.28456 [cs.AI]
(or arXiv:2605.28456v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.28456 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 285 words
Related coverage
Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model
arXiv · Jul 31, 2026
Best Open Speech Recognition (ASR) Models in 2026: WER, Languages, Latency, and License Compared
MarkTechPost · Jul 23, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
[AINews] Claude Opus 5.5, the new default model for AINews — and everybody cuts prices 40-50%
Latent Space · Sep 23, 2026