analiza aiqbench
CounterCount exposes a critical flaw in Vision-Language Models where they prioritize internal knowledge priors over actual visual evidence, causing accuracy to plummet when presented with counterfactual scenarios like an animal with extra limbs. The study reveals that models exhibit "System-1" reasoning by defaulting to canonical counts and underutilizing existing visual activations, suggesting that attention manipulation or reasoning cues are necessary to mitigate this bias. This diagnostic dataset provides a rigorous benchmark for evaluating how deeply VLMs truly perceive attributes versus merely recalling facts about them.
CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes
arXiv:2605.17826v2 Announce Type: replace-cross Abstract: Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly ($98.9\%$ and $97.5\%$), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from $91.6\%$ on factual to $49.5\%$ on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a ``System-1''-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.
Abstract:Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly ($98.9\%$ and $97.5\%$), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from $91.6\%$ on factual to $49.5\%$ on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a ``System-1''-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2605.17826 [cs.CV]
(or arXiv:2605.17826v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2605.17826 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 329 cuvinte