analiza aiqbench
This research introduces MME-Safety, a novel benchmark designed to rigorously evaluate the safety vulnerabilities of Multimodal Large Language Models (MLLMs) that often bypass unimodal filters. By utilizing a four-dimensional annotation schema and a hierarchical evaluation framework, the study exposes critical risks in cross-modal inputs and Chain-of-Thought reasoning across 17 state-of-the-art systems. The findings highlight an urgent need for robust, reasoning-aware safety alignment to ensure the structural integrity of defensive behaviors in emerging multimodal architectures.
MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
arXiv:2609.20850v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easily bypass unimodal filters. Existing benchmarks lack fine-grained intent-related annotations and rely on unidimensional metrics, hindering comprehensive robustness evaluation. To address this, we propose MME-Safety, a rigorously verified benchmark featuring a unique four-dimensional annotation schema that categorizes risk scenarios, harm severity, and modality-specific stealth levels. Furthermore, we introduce a hierarchical evaluation framework to assess fundamental response reliability, actual risk exposure, and the structural integrity of defensive behaviors. Extensive zero-shot evaluations across 17 state-of-the-art MLLMs provide a comprehensive safety profile of current multimodal systems. Our analysis systematically investigates cross-modal input configurations and uncovers safety implications associated with Chain-of-Thought (CoT) reasoning. These multifaceted findings underscore the urgent need for robust, reasoning-aware safety alignment in the multimodal landscape.
Abstract:While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easily bypass unimodal filters. Existing benchmarks lack fine-grained intent-related annotations and rely on unidimensional metrics, hindering comprehensive robustness evaluation. To address this, we propose MME-Safety, a rigorously verified benchmark featuring a unique four-dimensional annotation schema that categorizes risk scenarios, harm severity, and modality-specific stealth levels. Furthermore, we introduce a hierarchical evaluation framework to assess fundamental response reliability, actual risk exposure, and the structural integrity of defensive behaviors. Extensive zero-shot evaluations across 17 state-of-the-art MLLMs provide a comprehensive safety profile of current multimodal systems. Our analysis systematically investigates cross-modal input configurations and uncovers safety implications associated with Chain-of-Thought (CoT) reasoning. These multifaceted findings underscore the urgent need for robust, reasoning-aware safety alignment in the multimodal landscape.
Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2609.20850 [cs.CL]
(or arXiv:2609.20850v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.20850 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 232 cuvinte
Subiecte conexe
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
arXiv · 12 sept. 2026
From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs
arXiv · 7 aug. 2026
MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities
arXiv · 27 aug. 2026
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
arXiv · 9 sept. 2026