aiqbench analysis
This study demonstrates that appending interpretable acoustic tokens to prompts effectively enhances speech emotion recognition in audio language models without degrading their reliance on raw audio signals. The findings confirm that models remain robust against token perturbations while showing sensitivity to symbolic cue channels, offering a practical method for probing interpretability and grounded cue usage in affective computing.
Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition
arXiv:2606.07309v1 Announce Type: cross Abstract: Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
Abstract:Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.07309 [cs.SD]
(or arXiv:2606.07309v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2606.07309 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Titans-as-a-Layer: Test-Time Memory for Conversational Speech Emotion Recognition
arXiv · Jun 9, 2026
ALAS: An Automatic Latent Alignment Score for Audio Language Models
arXiv · Sep 30, 2026
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
arXiv · Aug 19, 2026
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
arXiv · Jun 9, 2026