aiqbench analysis
This paper addresses the critical train-evaluation gap in visual robot locomotion by introducing the Neverwhere Benchmark Suite, which utilizes over sixty 3D Gaussian Splatting reconstructions to simulate complex urban environments. The work provides essential policy checkpoints and analysis demonstrating that relying exclusively on synthetic Gaussian data can lead to performance degradation when deployed in novel scenes, highlighting the necessity for diverse training datasets.
The Neverwhere Visual Parkour Benchmark Suite
arXiv:2609.16443v1 Announce Type: new Abstract: State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world performance before deployment increasingly difficult. This work intends to narrow this train/evaluation gap by developing a collection of hyper-photo-realistic, closed-loop evaluation environments - The Neverwhere Benchmark Suite - comprised of over sixty 3D Gaussian Splatting reconstructions of urban indoor and outdoor scenes. Our goal is to encourage large-scale and reproducible robot evaluation by making it easier to create and integrate Gaussian splats-based reconstructions into simulated continuous testing setups. We also underscore the potential pitfalls of relying exclusively on 3D Gaussian-generated data for training, by providing policy checkpoints trained over multiple Neverwhere scenes and their performance when evaluated in novel scenes. Our analysis illustrates the necessity of sourcing diverse data to ensure performance. Code and data are available on the project page: https://ziyc.github.io/neverwhere-bench/.
Authors:Ziyu Chen, Henghui Bao, Haoran Chang, Alan Yu, Ran Choi, Kai McClennen, Gio Huh, Kevin Yang, Ri-Zhao Qiu, Yajvan Ravan, John J. Leonard, Xiaolong Wang, Phillip Isola, Ge Yang, Yue Wang
Abstract:State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world performance before deployment increasingly difficult. This work intends to narrow this train/evaluation gap by developing a collection of hyper-photo-realistic, closed-loop evaluation environments - The Neverwhere Benchmark Suite - comprised of over sixty 3D Gaussian Splatting reconstructions of urban indoor and outdoor scenes. Our goal is to encourage large-scale and reproducible robot evaluation by making it easier to create and integrate Gaussian splats-based reconstructions into simulated continuous testing setups. We also underscore the potential pitfalls of relying exclusively on 3D Gaussian-generated data for training, by providing policy checkpoints trained over multiple Neverwhere scenes and their performance when evaluated in novel scenes. Our analysis illustrates the necessity of sourcing diverse data to ensure performance. Code and data are available on the project page: this https URL.
9 pages, 14 figures. Accepted to IROS 2026. Project page: this https URL
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2609.16443 [cs.RO]
(or arXiv:2609.16443v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.16443 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 210 words
Related coverage
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Introducing Embed 5—A new family of frontier embedding models
Cohere · Sep 30, 2026
Beyond One-Size-Fits-All: Sample-Adaptive Strategy Routing for Vision Token Pruning in MLLMs
arXiv · Sep 11, 2026
How to Evaluate General-Purpose Robot Policies for Real-World Deployment
NVIDIA · Jul 12, 2026