aiqbench analysis
This research introduces pseudo-ensembles for music audio-language models, enabling them to abstain from answering when uncertain without the computational cost of training multiple independent models. By perturbing input features like answer order or audio corruption, the method generates diverse predictive distributions that improve accuracy and provide richer uncertainty metrics than single-pass entropy alone. This approach offers a practical pathway to reduce lucky guesses and better assess true musical understanding in compact AI systems.
Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models
arXiv:2609.04362v1 Announce Type: cross Abstract: Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks the same as real musical understanding. What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing. The usual solution, an ensemble of independently trained models, is far too expensive here, which leaves the entropy of a single predictive distribution as the only available confidence signal. We instead build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer, then averaging the resulting distributions over the options. Our main construction simply shuffles the order in which the candidate answers are presented; we also study ensembles built from corrupted audio and from swapped option labels. A pseudo-ensemble gives several predictive distributions per question, so it supports the full family of ensemble-based uncertainty measures (entropy of the expected distribution, expected entropy, and their difference, the mutual information) rather than entropy alone. Evaluating TinyMU on MuChoMusic, we find that averaging over four option orderings raises accuracy from 55.7% to 59.2%, and that the resulting uncertainty measures rank the model's errors better than the single-pass entropy baseline, reducing the area under the error retention curve from 0.293 to 0.261. All of this costs a few extra forward passes and no retraining, which makes abstention practical for compact music audio-language models.
Abstract:Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks the same as real musical understanding. What is missing is a way to tell when the model does not know the answer, so that it can abstain instead of guessing. The usual solution, an ensemble of independently trained models, is far too expensive here, which leaves the entropy of a single predictive distribution as the only available confidence signal. We instead build pseudo-ensembles from one pretrained model by perturbing its input in ways that cannot change the correct answer, then averaging the resulting distributions over the options. Our main construction simply shuffles the order in which the candidate answers are presented; we also study ensembles built from corrupted audio and from swapped option labels. A pseudo-ensemble gives several predictive distributions per question, so it supports the full family of ensemble-based uncertainty measures (entropy of the expected distribution, expected entropy, and their difference, the mutual information) rather than entropy alone. Evaluating TinyMU on MuChoMusic, we find that averaging over four option orderings raises accuracy from 55.7% to 59.2%, and that the resulting uncertainty measures rank the model's errors better than the single-pass entropy baseline, reducing the area under the error retention curve from 0.293 to 0.261. All of this costs a few extra forward passes and no retraining, which makes abstention practical for compact music audio-language models.
Sound (cs.SD); Computation and Language (cs.CL)
arXiv:2609.04362 [cs.SD]
(or arXiv:2609.04362v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.04362 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 338 words
Related coverage
Podcast with Harry Kumar, Co-founder, Director & CCO of Moth Quantum
Quantum Computing Report · Oct 9, 2026
How Sonos rebooted itself
The Verge · Sep 3, 2026
Never Post’s Mike Rugnetta on the creative process and the value of reliable power
The Verge · Jun 13, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026