analiza aiqbench
This research investigates whether current multimodal embedding models can accurately map textual descriptions of sound qualities—such as brightness or warmth—to their corresponding audio signals. The findings indicate that while LAION-CLAP demonstrates the strongest alignment with human-perceived timbre among tested architectures, existing embeddings still only partially capture these complex perceptual semantics. This work highlights critical gaps in how AI models represent auditory attributes, offering specific insights for improving text-guided music generation and audio retrieval systems.
Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?
arXiv:2510.14249v2 Announce Type: replace-cross Abstract: Understanding and modeling the relationship between language and sound are essential for applications such as music information retrieval, text-guided music generation, and audio captioning. Central to these tasks are joint language-audio embedding spaces, which map textual descriptions and auditory content into a shared representation. Although multimodal embedding models such as MS-CLAP, LAION-CLAP, MuQ-MuLan, and OpenFLAM have shown strong performance in language-audio alignment, their correspondence to human perception of timbre, a multifaceted attribute encompassing qualities such as brightness, roughness, and warmth, remains under-explored. In this paper, we evaluate these joint language-audio embedding models in terms of their ability to capture perceptual timbre semantics. Across two complementary experiments, we find that LAION-CLAP shows relatively strong and consistent alignment with human-perceived timbre semantics across both instrumental sounds and descriptor-conditioned audio effects. At the same time, the overall strength of this alignment remains limited, suggesting that current joint language-audio embeddings capture perceptual timbre semantics only partially. We also observe that, overall, reverb-induced timbre semantics are more consistently encoded than equalization-induced timbre semantics.
Title:Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?
Abstract:Understanding and modeling the relationship between language and sound are essential for applications such as music information retrieval, text-guided music generation, and audio captioning. Central to these tasks are joint language-audio embedding spaces, which map textual descriptions and auditory content into a shared representation. Although multimodal embedding models such as MS-CLAP, LAION-CLAP, MuQ-MuLan, and OpenFLAM have shown strong performance in language-audio alignment, their correspondence to human perception of timbre, a multifaceted attribute encompassing qualities such as brightness, roughness, and warmth, remains under-explored. In this paper, we evaluate these joint language-audio embedding models in terms of their ability to capture perceptual timbre semantics. Across two complementary experiments, we find that LAION-CLAP shows relatively strong and consistent alignment with human-perceived timbre semantics across both instrumental sounds and descriptor-conditioned audio effects. At the same time, the overall strength of this alignment remains limited, suggesting that current joint language-audio embeddings capture perceptual timbre semantics only partially. We also observe that, overall, reverb-induced timbre semantics are more consistently encoded than equalization-induced timbre semantics.
Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
arXiv:2510.14249 [cs.SD]
(or arXiv:2510.14249v2 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2510.14249 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 275 cuvinte
Subiecte conexe
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
arXiv · 6 iun. 2026
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
arXiv · 18 aug. 2026
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
arXiv · 9 iun. 2026
AuRA: Internalizing Audio Understanding into LLMs as LoRA
arXiv · 10 iun. 2026