aiqbench analysis
This research addresses "cross-modal safety drift" in multimodal large language models, where benign text queries paired with risky images bypass standard safety filters. The authors propose a lightweight representation transfer method that effectively mitigates this vulnerability without retraining the model backbone. For practitioners deploying vision-language systems, this work highlights a critical gap in current alignment techniques and offers a practical solution to harden models against visual-contextualized harms.
Transfer Safety Awareness for Cross-Modal Safety Drift in Multimodal Large Language Models
arXiv:2609.02082v1 Announce Type: cross Abstract: Visual modality enhances the capabilities of multimodal large language models (MLLMs) but also introduces a safety concern: a benign textual query may convey harmful intent when grounded in a visual image. We term this cross-modal safety drift and our pilot studies show that the safety response rate for such requests is substantially lower than that for requests containing explicitly unsafe text. This paper aims to systematically study this issue. First, we conduct an empirical analysis to identify representative unsafe response patterns. Building on these, we interpret model representations and attentions, revealing that visually risky cues receive limited attention and weakly trigger refusal. Motivated by the observation that safety signals from unsafe text processing can be transferred, we propose safety-awareness representation transfer (SRT), a lightweight direction-refinement method that mitigates cross-modal safety drift with a frozen MLLM backbone. Experiments across multiple benchmarks and models show that SRT effectively improves safety in diverse cross-modal settings while preserving utility. Code is available at https://github.com/cucu220123/safety-awareness.
Abstract:Visual modality enhances the capabilities of multimodal large language models (MLLMs) but also introduces a safety concern: a benign textual query may convey harmful intent when grounded in a visual image. We term this cross-modal safety drift and our pilot studies show that the safety response rate for such requests is substantially lower than that for requests containing explicitly unsafe text. This paper aims to systematically study this issue. First, we conduct an empirical analysis to identify representative unsafe response patterns. Building on these, we interpret model representations and attentions, revealing that visually risky cues receive limited attention and weakly trigger refusal. Motivated by the observation that safety signals from unsafe text processing can be transferred, we propose safety-awareness representation transfer (SRT), a lightweight direction-refinement method that mitigates cross-modal safety drift with a frozen MLLM backbone. Experiments across multiple benchmarks and models show that SRT effectively improves safety in diverse cross-modal settings while preserving utility. Code is available at this https URL.
Multimedia (cs.MM); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR)
arXiv:2609.02082 [cs.MM]
(or arXiv:2609.02082v1 [cs.MM] for this version)
https://doi.org/10.48550/arXiv.2609.02082 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
Related coverage
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
arXiv · Oct 7, 2026
FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities
arXiv · Jul 28, 2026
CCMAN: Cognitive Instability-Aware Cross-Modal Attention Network for Interpretable Temporal Biomarkers of Verbal Fluency Speech
arXiv · Sep 15, 2026
DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding
arXiv · Sep 3, 2026