aiqbench analysis
This research critically evaluates the CLAP metric used in text-to-music generation, revealing that current models often rely on audio-agnostic language priors rather than true semantic understanding of musical attributes. The study demonstrates that these embeddings function more like bag-of-words representations, failing to distinguish fine-grained details when specific instrument properties are swapped in captions. Consequently, the standard objective metric for faithfulness is shown to be insufficient for capturing the nuanced meaning required by advanced audio-language models.
Don't CLAP: Are Music-Text Models Bag-of-Words?
arXiv:2609.30540v1 Announce Type: cross Abstract: Text-to-music systems are assessed on audio quality and on how faithfully the music follows its prompt, and the CLAP score, the cosine similarity between a music-text model's audio and text embeddings, is the standard objective metric of faithfulness. We ask how accurately that score reflects the text: when an attribute is linked to an instrument (e.g., distorted guitar), does the text embedding capture that binding? To find out, we introduce an attribute swap perturbation: the caption of a real recording is edited by exchanging exactly one property, timbre, lead versus accompaniment, or order of first appearance, between two instruments. We then test four contrastive music-text models and one large audio-language model on whether the audio scores higher against the original caption than against the perturbed one. No contrastive model distinguishes the two captions reliably. The audio-language model does better, but further experiments show that its advantage rests largely on audio-agnostic language priors. Our results thus provide compelling evidence that the CLAP score and related metrics do not capture fine-grained musical meaning or attribute bindings; their representation is closer to a bag-of-words that leaves them insensitive to meaning-changing perturbations of the caption.
Title:Don't CLAP: Are Music-Text Models Bag-of-Words?
Abstract:Text-to-music systems are assessed on audio quality and on how faithfully the music follows its prompt, and the CLAP score, the cosine similarity between a music-text model's audio and text embeddings, is the standard objective metric of faithfulness. We ask how accurately that score reflects the text: when an attribute is linked to an instrument (e.g., distorted guitar), does the text embedding capture that binding? To find out, we introduce an attribute swap perturbation: the caption of a real recording is edited by exchanging exactly one property, timbre, lead versus accompaniment, or order of first appearance, between two instruments. We then test four contrastive music-text models and one large audio-language model on whether the audio scores higher against the original caption than against the perturbed one. No contrastive model distinguishes the two captions reliably. The audio-language model does better, but further experiments show that its advantage rests largely on audio-agnostic language priors. Our results thus provide compelling evidence that the CLAP score and related metrics do not capture fine-grained musical meaning or attribute bindings; their representation is closer to a bag-of-words that leaves them insensitive to meaning-changing perturbations of the caption.
Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
arXiv:2609.30540 [cs.SD]
(or arXiv:2609.30540v1 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2609.30540 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 299 words
Related coverage
AI Futures: Racing to Lose, A sermon
LessWrong · Sep 28, 2026
How robust are natural language autoencoders to initialization?
AI Alignment Forum · Jul 10, 2026
Nicole’s guide to writing and editing
Quantum Frontiers · Aug 26, 2026
Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax
LessWrong · Oct 9, 2026