analiza aiqbench
This study isolates the bottleneck in audio-conditioned LLMs, demonstrating that acoustic information loss stems primarily from readout alignment failures rather than inherent limitations of speech recognition frontends. By tracing discriminative structure through model layers, the research reveals that standard decoding procedures and answer formats often suppress recoverable prosody and environmental cues. These findings provide a critical diagnostic for improving multimodal models to better utilize non-speech acoustic data.
Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
arXiv:2609.05871v1 Announce Type: cross Abstract: Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Title:Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
Abstract:Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
Sound (cs.SD); Artificial Intelligence (cs.AI)
arXiv:2609.05871 [cs.SD]
(or arXiv:2609.05871v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.05871 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 286 cuvinte
Subiecte conexe
Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models
arXiv · 2 sept. 2026
Six Principles of Voice Design: Designing for the Ear and Encoding It into Agent Script
Salesforce Blog · 8 oct. 2026
Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck
arXiv · 17 oct. 2026
Physics-Informed Neural Engine Sound Modeling with Differentiable Pulse-Train Synthesis
arXiv · 4 iun. 2026