aiqbench analysis
MCBench introduces a critical evaluation framework for Omni LLMs by testing their ability to integrate vision, audio, and text for safety judgments, revealing that current models struggle with subtle, non-physical risks despite strong performance on salient cues. The study highlights a fundamental gap in cross-modal reasoning where models fail to effectively synthesize extracted information for safety-critical decisions. This finding underscores the necessity for architectural improvements and training strategies specifically designed to enhance robustness in multimodal safety scenarios.
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
arXiv:2606.05177v1 Announce Type: cross Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for multimodal safety.
Abstract:Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for multimodal safety.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
arXiv:2606.05177 [cs.CL]
(or arXiv:2606.05177v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.05177 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 236 words
Related coverage
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
arXiv · Aug 18, 2026
[AINews] Modal’s $1B ARR, $16B Series D makes it NYC’s newest AI Infra Decacorn, 3x usage and 3x valuation in 5 months, 15x in 1 yr
Latent Space · Oct 15, 2026
Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models
arXiv · Aug 25, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026