VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

Title:VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

Abstract:Visual Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping Vision-Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual retrieval-augmented framework for multi-image reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and reasoning. Experiments on multiple visual question answering benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19\%, while substantially reducing visual hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and reasoning reliability in multi-image settings. Codes and data are available at this https URL

Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)

arXiv:2510.09733 [cs.CL]

(or arXiv:2510.09733v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2510.09733 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.