Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

Title:Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

Abstract:Vision-language models (VLMs) are increasingly applied to medical visual question answering (Med-VQA), yet whether they can \emph{localize} the evidence behind their answers---a prerequisite for clinical auditability---is poorly characterized. We separately evaluate VQA reasoning and visual grounding for four recent frontier VLMs (GPT-5.1, GPT-5.5, Gemini-2.5-Pro, Gemini-3-Flash), two domain-specific medical VLMs (Lingshu, MedGemma), and a dedicated open-vocabulary detector (Grounding DINO) on VQA-RAD and SLAKE. Two findings challenge the intuition that ``add grounding to improve VQA.'' First, \textbf{no model localizes medical targets well}: every off-the-shelf system---frontier, medical-specialized, or dedicated detector---scores mean IoU 0.05--0.24 on our SLAKE grounding split, at or barely above a trivial center-box baseline (0.10), with Acc@0.5 below 20\%. Second, and counter to the common ``localize-then-answer'' paradigm, \textbf{cropping to a bounding box degrades VQA even when the box is a perfect oracle}: on the matched subset where oracle ground-truth boxes are applied, GT-grounding \emph{lowers} closed-ended accuracy for every model (by 0.9--18.0 points versus using the full image)---consistent with the crop discarding global context the model relies on. Because the oracle box removes localization error by construction, the problem is not that perception is a recoverable bottleneck, but that grounding-by-cropping is itself the wrong interface. Finally, we show constructively that the two channels need not conflict: supervised fine-tuning of Qwen-2.5-VL-7B on answers \emph{alone} silently destroys box-evidence emission (0/418 parseable boxes), whereas mixing in a small amount of grounding supervision restores localization to 0.36 IoU---above every zero-shot model---while preserving answer accuracy.

Artificial Intelligence (cs.AI)

arXiv:2604.27720 [cs.AI]

(or arXiv:2604.27720v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2604.27720 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.