aiqbench analysis
This research applies concept bottleneck models to speech emotion recognition (SER) using large language models to improve explainability and reduce bias toward text transcripts. By isolating acoustic concepts like speech rate and intensity, the study reveals that aggregate performance metrics often mask significant shifts in individual prediction logic when specific audio features are removed. These findings offer critical insights for developers seeking to build transparent, multi-modal AI systems that accurately interpret human emotion without over-relying on textual cues.
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
arXiv:2609.39453v1 Announce Type: cross Abstract: Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.
Abstract:Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.
5 pages, 2 figures. Submitted to ICASSP 2027
Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
arXiv:2609.39453 [cs.SD]
(or arXiv:2609.39453v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.39453 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 303 words
Related coverage
Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition
arXiv · Jun 8, 2026
SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows
arXiv · Sep 29, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
arXiv · Aug 19, 2026