aiqbench analysis
This study benchmarks Visual Language Models (VLMs) against human drivers in challenging, out-of-distribution environments like Lima and New York City to assess generalization capabilities for autonomous driving systems. The findings reveal that while VLM responses diverge from humans based on question type, geographic origin has minimal impact on performance due to the extreme novelty of the test scenarios. This research provides critical insights into the robustness of multi-modal cognitive backbones when deployed in international markets where no current self-driving operations exist.
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
arXiv:2606.20980v2 Announce Type: replace-cross Abstract: As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2
Abstract:As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: this https URL
Accepted as a Workshop paper at the Drive-X workshop (Foundation Models for Autonomous Driving) at the European Conference in Computer Vision (ECCV), 2026. Malmo, Sweden. 17 pages main body. 51 pages total. Data publicly available online
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO)
arXiv:2606.20980 [cs.CV]
(or arXiv:2606.20980v2 [cs.CV] for this version)
Full text under the publisher's open licence · 253 words
Related coverage
34 Amazon Research Awards Build on Trainium recipients announced
Amazon Science · Aug 5, 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · Jul 26, 2026
From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
arXiv · Jun 4, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026