aiqbench analysis
This research investigates whether geometric alignment between speech and text representations in large language models guarantees functional equivalence, finding that mid-layer alignment does not necessarily lead to accurate answers. By analyzing layer-wise answer margins and attention mechanisms across four open-weight models, the study identifies specific failure modes where speech inputs yield lower accuracy despite semantic matching. The findings suggest that compressing speech signals can repair certain errors, offering a potential pathway to improve modality consistency in end-to-end speech LLMs.
Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap
arXiv:2603.01502v2 Announce Type: replace Abstract: End-to-end speech large language models (LLMs) often answer the same question less accurately from speech than from text, even when the spoken content matches the text prompt. Existing analyses study this modality gap through the geometry of representations and measure how closely speech representations align with text. Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way. This work connects representation geometry with answer formation and asks whether geometric alignment implies functional equivalence. We compare cross-layer Centered Kernel Alignment (CKA) with layer-wise answer margins for four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH. In Qwen2.5-Omni and MiniCPM-o, speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer alignment does not imply functional equivalence. Qwen2-Audio instead shows weak alignment, which indicates a different failure mode. An attention intervention at fixed input and token count identifies a causal effect of late attention allocation on answer-code separation, a quantity distinct from answer correctness. Compressing the speech signal or its token sequence repairs a subset of the errors that occur only with speech.
Abstract:End-to-end speech large language models (LLMs) often answer the same question less accurately from speech than from text, even when the spoken content matches the text prompt. Existing analyses study this modality gap through the geometry of representations and measure how closely speech representations align with text. Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way. This work connects representation geometry with answer formation and asks whether geometric alignment implies functional equivalence. We compare cross-layer Centered Kernel Alignment (CKA) with layer-wise answer margins for four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH. In Qwen2.5-Omni and MiniCPM-o, speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer alignment does not imply functional equivalence. Qwen2-Audio instead shows weak alignment, which indicates a different failure mode. An attention intervention at fixed input and token count identifies a causal effect of late attention allocation on answer-code separation, a quantity distinct from answer correctness. Compressing the speech signal or its token sequence repairs a subset of the errors that occur only with speech.
Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
arXiv:2603.01502 [cs.CL]
(or arXiv:2603.01502v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.01502 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
Game Design Creativity evals
LessWrong · Oct 12, 2026
How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago
LessWrong · Jul 22, 2026
Research directions in condensation: varieties of objectivity
AI Alignment Forum · Jul 28, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026