aiqbench analysis
This arXiv paper proposes a two-stage neural network for audio-visual speech enhancement, using lip or visual cues to separate a target speaker from noisy, reverberant mixtures. It reports top-tier results on the ISCSLP 2026 Real-World AVSE Challenge, with the lowest character error rate on both tracks. For AI readers, the notable part is the coarse-to-fine design and how much visual conditioning helps in real-world, multi-speaker conditions, though the results are challenge-specific and still need broader validation.
CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge
arXiv:2610.21773v1 Announce Type: cross Abstract: Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage AVSE framework. In the first stage, a progressive SPAV-TFGridNet is extended to two synchronized visual streams to produce visually ordered coarse estimates by progressively suppressing interfering speech while preserving target reverberation and shared background noise. In the second stage, a TFGridNet-based target speaker refinement network takes the original mixture and both coarse estimates as input to further recover target speech and improve signal quality. The coarse and refined estimates are finally combined using development-set-calibrated fusion. On the ISCSLP 2026 Real-World AVSE Challenge, our system tied for first place on Track 1 and ranked second on Track 2 in OVRL, while achieving the lowest CER on both tracks at 12.31% and 14.79%, respectively.
Abstract:Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage AVSE framework. In the first stage, a progressive SPAV-TFGridNet is extended to two synchronized visual streams to produce visually ordered coarse estimates by progressively suppressing interfering speech while preserving target reverberation and shared background noise. In the second stage, a TFGridNet-based target speaker refinement network takes the original mixture and both coarse estimates as input to further recover target speech and improve signal quality. The coarse and refined estimates are finally combined using development-set-calibrated fusion. On the ISCSLP 2026 Real-World AVSE Challenge, our system tied for first place on Track 1 and ranked second on Track 2 in OVRL, while achieving the lowest CER on both tracks at 12.31% and 14.79%, respectively.
Sound (cs.SD); Artificial Intelligence (cs.AI)
arXiv:2610.21773 [cs.SD]
(or arXiv:2610.21773v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2610.21773 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 233 words
Related coverage
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
arXiv · Jun 9, 2026
NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware
arXiv · Sep 22, 2026
GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model
arXiv · Jun 9, 2026
NVIDIA Brings Real-Time AI to Broadcast, Sports and Global Streaming at IBC
NVIDIA · Sep 9, 2026