aiqbench analysis
This research addresses the critical "mirage" failure mode in vision-language models, where systems confidently hallucinate visual evidence that is absent or irrelevant to a query. By introducing Text-Conditioned Layer-wise Internal Alignment (TC-LIA), the authors present an ensemble method capable of detecting these ungrounded answers before they reach users, significantly improving safety for high-stakes applications like medical imaging and document analysis. The study demonstrates substantial reductions in hallucination rates across fourteen different VLM architectures without requiring model retraining.
Detect Before You Leap: Mirage Detection in Vision-Language Models
arXiv:2606.00435v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question. This failure mode, recently described as mirage (Asadi et al., 2026), is especially concerning in medical and document VQA, where visually ungrounded answers may be mistaken for image-based evidence. We study pre-release mirage detection: given an image-question pair, determine whether a VLM's answer should be released or the system should abstain before the answer reaches the user. We propose Text-Conditioned Layer-wise Internal Alignment (TC-LIA), a model-agnostic method that probes patch-token representations across the layers of a CLIP ViT-H/14 vision encoder. The key idea is to project layer-wise image patch tokens into the final CLIP embedding space and measure their similarity with the question embedding, tracking whether question-relevant visual evidence emerges across vision layers. TC-LIA summarizes this trajectory using final image-text cosine similarity, late-layer top-k patch-text alignment, early-to-late gain, and layer-wise slope. These features are combined with pixel-statistic-based blank/noise detection, zero-shot domain routing, and structured VLM self-assessment in an ensemble classifier. We evaluate on 19,004 samples spanning ten VQA domains, including medical imaging, document understanding, scene text, reasoning, and video understanding, with four input conditions: RELATED, IN-DOMAIN UNRELATED, OUT-OF-DOMAIN UNRELATED, and BLANK/NOISE. Across fourteen VLMs, base prompt mirage rates range from 57.3-75.0%. TC-LIA alone reduces the mirage rate to 7.5% at 83.5% accuracy; ensemble fusion further improves performance to 84.5-88.4% accuracy with 5.7-7.2% mirage rates. The best result, obtained with Qwen2.5-VL-72B, achieves 88.4% accuracy and a 6.4% mirage rate.
Abstract:Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question. This failure mode, recently described as mirage (Asadi et al., 2026), is especially concerning in medical and document VQA, where visually ungrounded answers may be mistaken for image-based evidence. We study pre-release mirage detection: given an image-question pair, determine whether a VLM's answer should be released or the system should abstain before the answer reaches the user. We propose Text-Conditioned Layer-wise Internal Alignment (TC-LIA), a model-agnostic method that probes patch-token representations across the layers of a CLIP ViT-H/14 vision encoder. The key idea is to project layer-wise image patch tokens into the final CLIP embedding space and measure their similarity with the question embedding, tracking whether question-relevant visual evidence emerges across vision layers. TC-LIA summarizes this trajectory using final image-text cosine similarity, late-layer top-k patch-text alignment, early-to-late gain, and layer-wise slope. These features are combined with pixel-statistic-based blank/noise detection, zero-shot domain routing, and structured VLM self-assessment in an ensemble classifier. We evaluate on 19,004 samples spanning ten VQA domains, including medical imaging, document understanding, scene text, reasoning, and video understanding, with four input conditions: RELATED, IN-DOMAIN UNRELATED, OUT-OF-DOMAIN UNRELATED, and BLANK/NOISE. Across fourteen VLMs, base prompt mirage rates range from 57.3-75.0%. TC-LIA alone reduces the mirage rate to 7.5% at 83.5% accuracy; ensemble fusion further improves performance to 84.5-88.4% accuracy with 5.7-7.2% mirage rates. The best result, obtained with Qwen2.5-VL-72B, achieves 88.4% accuracy and a 6.4% mirage rate.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2606.00435 [cs.CV]
(or arXiv:2606.00435v3 [cs.CV] for this version)
From: Md. Shaown Miah [view email] [v1] Fri, 29 May 2026 23:51:35 UTC (11,848 KB) [v2] Mon, 15 Jun 2026 15:09:01 UTC (13,483 KB) [v3] Mon, 27 Jul 2026 23:31:43 UTC (16,234 KB)
Full text under the publisher's open licence · 300 words
Related coverage
Detect Before You Leap: Mirage Detection in Vision-Language Models
arXiv · Jun 16, 2026
NVIDIA and Japan Bring Full-Stack AI and Robotics to Every Industry
NVIDIA · Jul 15, 2026
MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents
arXiv · Oct 17, 2026
Build a Multi-Camera 3D Tracking Application with NVIDIA DeepStream 9.1 Skills
NVIDIA · Jul 15, 2026