aiqbench analysis
This study reveals a critical architectural gap in audio-language models where paralinguistic cues like tone are heavily encoded but systematically degraded before reaching the output layer. The analysis demonstrates that current decoders either prioritize text content or acoustic style, with few successfully preserving both simultaneously despite strong initial encoding. This finding highlights a fundamental limitation in how these models process and utilize non-verbal speech information for downstream tasks.
Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models
arXiv:2609.00727v1 Announce Type: cross Abstract: Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.
Abstract:Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.
Sound (cs.SD); Artificial Intelligence (cs.AI)
arXiv:2609.00727 [cs.SD]
(or arXiv:2609.00727v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.00727 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 298 words
Related coverage
Six Principles of Voice Design: Designing for the Ear and Encoding It into Agent Script
Salesforce Blog · Oct 8, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
arXiv · Sep 9, 2026
FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
arXiv · Aug 26, 2026