analiza aiqbench
This research exposes a critical vulnerability in Vision-Language Models where they prioritize semantic plausibility over literal accuracy when scene context conflicts with printed text. The introduction of the SceneFaith benchmark demonstrates that current models frequently rewrite clear text to match environmental cues, highlighting the need for architectures that strictly separate visual evidence from contextual inference. For developers building reliable AI systems, this underscores the necessity of balancing character-level precision with contextual understanding to prevent hallucinations in real-world scenarios.
When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
arXiv:2609.34781v1 Announce Type: cross Abstract: Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45\% to 58.51\%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.
Abstract:Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45\% to 58.51\%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2609.34781 [cs.CV]
(or arXiv:2609.34781v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.34781 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 266 cuvinte
Subiecte conexe
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · 21 mai 2026
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
arXiv · 23 iul. 2026
AI-generated Images Challenge Visual Trust in High-risk Scenarios
arXiv · 28 iul. 2026
GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts
arXiv · 28 aug. 2026