aiqbench analysis
This research introduces Omni-Embed-Audio, a retrieval encoder that leverages multimodal LLMs to significantly improve robustness against natural user queries and hard acoustic distractors compared to traditional CLAP-based systems. By shifting evaluation from caption-style inputs to diverse User-Intent Queries, the study demonstrates that LLM backbones offer superior semantic understanding for complex audio-text search tasks. The work highlights a critical gap in current benchmarks where real-world search behaviors are often overlooked in favor of simplified caption matching.
Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval
arXiv:2604.18360v4 Announce Type: replace-cross Abstract: Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, these benchmarks rely on caption-style queries that differ substantially from real-world search behavior, limiting their assessment of practical retrieval robustness. We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding. To systematically evaluate robustness beyond caption-style queries, we introduce User-Intent Queries (UIQs) - five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. For negative queries, we develop a hard negative mining pipeline and propose discrimination metrics (HNSR, TFR) assessing models' ability to suppress acoustically similar distractors. Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP, while demonstrating clear advantages in two critical areas: (1) dominant text-to-text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10), revealing that LLM backbones provide superior semantic understanding of complex queries.
Abstract:Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, these benchmarks rely on caption-style queries that differ substantially from real-world search behavior, limiting their assessment of practical retrieval robustness. We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding. To systematically evaluate robustness beyond caption-style queries, we introduce User-Intent Queries (UIQs) - five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. For negative queries, we develop a hard negative mining pipeline and propose discrimination metrics (HNSR, TFR) assessing models' ability to suppress acoustically similar distractors. Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP, while demonstrating clear advantages in two critical areas: (1) dominant text-to-text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10), revealing that LLM backbones provide superior semantic understanding of complex queries.
Accepted at ACL 2026 Main Conference. Final version
Sound (cs.SD); Computation and Language (cs.CL)
arXiv:2604.18360 [cs.SD]
(or arXiv:2604.18360v4 [cs.SD] for this version)
https://doi.org/10.48550/arXiv.2604.18360 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 260 words
Related coverage
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
arXiv · Jun 6, 2026
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
arXiv · Aug 18, 2026
How to Build In-Vehicle AI Agents with NVIDIA: From Cloud to Car
NVIDIA · May 5, 2026