aiqbench analysis
This research addresses a critical limitation in audio-language models where they rely too heavily on textual shortcuts rather than actual acoustic evidence. The proposed Reward-Tilted On-Policy Distillation method introduces a specific reward mechanism to force compact student models to prioritize genuine audio inputs during training. Results demonstrate that this approach significantly improves acoustic grounding, allowing smaller 3B parameter models to outperform larger counterparts on standard benchmarks.
Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models
arXiv:2609.28778v1 Announce Type: cross Abstract: Audio-language models (ALMs) can exploit textual shortcuts to answer questions while overlooking acoustic evidence, weakening audio understanding. On-policy distillation (OPD) trains compact ALMs by supervising student-generated responses with teacher predictions, but does not explicitly distinguish acoustic support from linguistic predictability. We propose Reward-Tilted On-Policy Distillation (RT-OPD) to strengthen acoustic grounding. Given the same question and student-generated text, a frozen teacher predicts the next token with and without audio inputs. Their log-probability contrast defines a reward that reshapes the teacher distribution for reverse-KL distillation, emphasizing the additional evidence provided by audio. Across two compact students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD. Experiments with silenced and replacement audio further suggest that RT-OPD strengthens the student's reliance on acoustic evidence. Our 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models. Code and model checkpoints are available at https://github.com/KaiyangLi1992/RT-OPD.
Abstract:Audio-language models (ALMs) can exploit textual shortcuts to answer questions while overlooking acoustic evidence, weakening audio understanding. On-policy distillation (OPD) trains compact ALMs by supervising student-generated responses with teacher predictions, but does not explicitly distinguish acoustic support from linguistic predictability. We propose Reward-Tilted On-Policy Distillation (RT-OPD) to strengthen acoustic grounding. Given the same question and student-generated text, a frozen teacher predicts the next token with and without audio inputs. Their log-probability contrast defines a reward that reshapes the teacher distribution for reverse-KL distillation, emphasizing the additional evidence provided by audio. Across two compact students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD. Experiments with silenced and replacement audio further suggest that RT-OPD strengthens the student's reliance on acoustic evidence. Our 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models. Code and model checkpoints are available at this https URL.
Sound (cs.SD); Computation and Language (cs.CL)
arXiv:2609.28778 [cs.SD]
(or arXiv:2609.28778v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.28778 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 249 words
Related coverage
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
arXiv · Jul 24, 2026
Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
arXiv · Aug 6, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Small yet Assistive: Spatially-Aware Post-Training for Low Vision
arXiv · Sep 25, 2026