analiza aiqbench
This study introduces 4MT-VLM, a benchmark evaluating how visual language models maintain spatial awareness when viewpoint cues change. Results reveal that even frontier models struggle to form stable 3D cognitive maps beyond short distances, falling significantly behind human performance in recognizing locations from unseen angles. The findings highlight a critical gap in current AI's ability to navigate dynamic environments without relying on static appearance features.
4MT-VLM: How Coarse Is a VLMs Cognitive Map?
arXiv:2609.39238v1 Announce Type: new Abstract: An agent that moves must recognise a place from a viewpoint it has never seen. We introduce 4MT-VLM, a dataset of procedurally generated landscapes, each rendered across five stimulus modes that remove appearance cues while holding layout fixed: shape and colour, shape only, colour only, bare terrain peaks with no objects, and a valley viewpoint that puts the peaks on the horizon. The last condition is commonly used in clinics to probe hippocampal function in human patients. We test this benchmark across sixteen different open and closed-source models and report 4AFC performance, a measure which is also used to grade human participants. We observe that models identify a place from the studied viewpoint but lose it once the camera moves, dropping below the 25% chance level at 135{\deg} where a human observer scores 85%. Frontier models (Gemini 3.8 Flash, GPT-5.6) answer only 39% and 31% of rotated trials correctly, recovering to 85% and 55% only when distractors are moved more than 30 meters apart. Our benchmark demonstrates that while current VLMs possess rudimentary cognitive maps, their spatial resolution remains fundamentally too coarse to maintain a stable, 3D understanding of the world once the viewpoint changes.
Title:4MT-VLM: How Coarse Is a VLMs Cognitive Map?
Abstract:An agent that moves must recognise a place from a viewpoint it has never seen. We introduce 4MT-VLM, a dataset of procedurally generated landscapes, each rendered across five stimulus modes that remove appearance cues while holding layout fixed: shape and colour, shape only, colour only, bare terrain peaks with no objects, and a valley viewpoint that puts the peaks on the horizon. The last condition is commonly used in clinics to probe hippocampal function in human patients. We test this benchmark across sixteen different open and closed-source models and report 4AFC performance, a measure which is also used to grade human participants. We observe that models identify a place from the studied viewpoint but lose it once the camera moves, dropping below the 25% chance level at 135° where a human observer scores 85%. Frontier models (Gemini 3.8 Flash, GPT-5.6) answer only 39% and 31% of rotated trials correctly, recovering to 85% and 55% only when distractors are moved more than 30 meters apart. Our benchmark demonstrates that while current VLMs possess rudimentary cognitive maps, their spatial resolution remains fundamentally too coarse to maintain a stable, 3D understanding of the world once the viewpoint changes.
Computation and Language (cs.CL)
arXiv:2609.39238 [cs.CL]
(or arXiv:2609.39238v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.39238 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 298 cuvinte
Subiecte conexe
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
arXiv · 6 iun. 2026
Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs
arXiv · 7 oct. 2026
Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors
arXiv · 11 sept. 2026
PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
arXiv · 5 iun. 2026