analiza aiqbench
This paper proposes a diagnostic for whether vision-language models actually use the image when they answer, by blanking the image and checking whether the model notices, whether its output changes, and whether it still gives a content answer. Its main finding is that a large share of answers on blank input are fabricated "sycophantic" responses, and that scaling within a model family often increases this behavior, while the newer Qwen3-VL generation largely avoids it. Readers should note that v5 substantially revised the headline numbers after a flaw in the alignment test was corrected, and that a blank-image probe is an indirect measure of visual reliance, so the framework's validity rests on the blinded human study.
To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
arXiv:2603.18373v5 Announce Type: replace-cross Abstract: When Vision-Language Models (VLMs) answer correctly, do they rely on the image? We introduce a Tri-Layer Diagnostic Framework that asks three questions of every (image, question) pair when the image is blanked: does the model register that the image is missing (Latent Anomaly Detection, LAD), does its output distribution depend on the image (Visual Necessity Score, VNS), and does it nonetheless commit to a content answer? The cascade assigns each pair of any existing benchmark to one of four modes: Perceptual Blindness, Language Shortcut, Visual Sycophancy (the blank image registers and matters, yet the model commits to a fabricated answer), or Robust Refusal. Across nine open-weight VLMs and 9,000 model-sample pairs, 28.1% are Visual Sycophancy, 18.6% Language Shortcut, 5.6% Perceptual Blindness, and 47.8% Robust Refusal. The modes are strongly model-specific: the Qwen3-VL models refuse on blank input in 93-98% of pairs, whereas Pixtral-12B commits to an answer in 72%. Within-generation scaling increases Visual Sycophancy in both Qwen2.5-VL (7B->72B: 25.5->40.1%) and Llama-3.2-Vision (11B->90B: 34.3->48.0%), while the newer Qwen3-VL generation nearly eliminates it. A blinded human study confirms that LAD and VNS track what annotators observe, and that 80% of Visual-Sycophancy responses carry no textual sign that the image was blank.
Abstract:When Vision-Language Models (VLMs) answer correctly, do they rely on the image? We introduce a Tri-Layer Diagnostic Framework that asks three questions of every (image, question) pair when the image is blanked: does the model register that the image is missing (Latent Anomaly Detection, LAD), does its output distribution depend on the image (Visual Necessity Score, VNS), and does it nonetheless commit to a content answer? The cascade assigns each pair of any existing benchmark to one of four modes: Perceptual Blindness, Language Shortcut, Visual Sycophancy (the blank image registers and matters, yet the model commits to a fabricated answer), or Robust Refusal. Across nine open-weight VLMs and 9,000 model-sample pairs, 28.1% are Visual Sycophancy, 18.6% Language Shortcut, 5.6% Perceptual Blindness, and 47.8% Robust Refusal. The modes are strongly model-specific: the Qwen3-VL models refuse on blank input in 93-98% of pairs, whereas Pixtral-12B commits to an answer in 72%. Within-generation scaling increases Visual Sycophancy in both Qwen2.5-VL (7B->72B: 25.5->40.1%) and Llama-3.2-Vision (11B->90B: 34.3->48.0%), while the newer Qwen3-VL generation nearly eliminates it. A blinded human study confirms that LAD and VNS track what annotators observe, and that 80% of Visual-Sycophancy responses carry no textual sign that the image was blank.
v5: corrects the alignment test of earlier versions (v1-v4), which counted explicit refusals as Visual Sycophancy; Visual Sycophancy is 28.1% (not 75.9%) and Robust Refusal 47.8% (not 0%); see Sec. 6. 21 pages
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2603.18373 [cs.CV]
(or arXiv:2603.18373v5 [cs.CV] for this version)
Text integral sub licența deschisă a publicației · 249 cuvinte
Subiecte conexe
Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax
LessWrong · 9 oct. 2026
We're talking past our models; or, How a model defined its "evil" vector as dread
LessWrong · 20 iul. 2026
How Identity and Opinion Shape Political Sycophancy in LLMs
arXiv · 1 sept. 2026
Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
arXiv · 14 oct. 2026