aiqbench analysis
This research identifies Feed-Forward Network (FFN) modules as the primary source of language-prior dominance in Large Vision-Language Models, offering a training-free mitigation strategy called FADE. By attenuating FFN outputs, the method effectively reduces hallucinations without sacrificing inference efficiency, addressing a critical reliability issue in multimodal AI systems.
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
arXiv:2606.29431v5 Announce Type: replace Abstract: Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.
Abstract:Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.
18 pages, 5 figures, 27 tables. Corrected author list; Yichen Guo, Kai Tang, and Jinhao You contributed equally
Artificial Intelligence (cs.AI)
arXiv:2606.29431 [cs.AI]
(or arXiv:2606.29431v5 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.29431 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 251 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 18, 2026
Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
arXiv · Jun 9, 2026
Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
arXiv · Jun 15, 2026
[AINews] Gemini 4 Argon: GDM’s answer to Astra/Fable, with 1M output
Latent Space · Oct 1, 2026