aiqbench analysis
This research introduces Attention-Steerable Contrastive Decoding (ASCD), a training-free method that directly manipulates cross-modal attention scores to suppress hallucinations in multimodal large language models. By amplifying stable text-centric heads and dampening spurious visual tokens, ASCD reduces generation errors by up to 38% across five different model backbones without adding computational overhead. The work establishes attention steering as a principled approach for enhancing the reliability of vision-language systems during inference.
ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
arXiv:2506.14766v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.
Abstract:Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
arXiv:2506.14766 [cs.CV]
(or arXiv:2506.14766v3 [cs.CV] for this version)
https://doi.org/10.1609/aaai.v40i12.38000 DOI(s) linking to related resources
Full text under the publisher's open licence · 178 words
Related coverage
Attention at Rest Stays at Rest: Breaking Visual Inertia to Mitigate Relation Hallucinations
arXiv · Oct 8, 2026
[AINews] Gemini 4 Argon: GDM’s answer to Astra/Fable, with 1M output
Latent Space · Oct 1, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
Mitigating Video Hallucinations via Attention-Guided Representation Perturbation
arXiv · Oct 15, 2026