analiza aiqbench
InsightMap advances embodied AI by integrating top-down spatial maps as explicit memory within a multimodal framework, enabling robots to navigate and reason using partial observations. The approach demonstrates significant gains in success rates for unseen environments and outperforms existing models on real-world robotic platforms like the Unitree Go2. This work highlights a critical step toward robust autonomous agents capable of situated reasoning and 3D grounding in dynamic settings.
InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning
arXiv:2609.37187v1 Announce Type: cross Abstract: Language-guided navigation requires connecting partial observations to a persistent spatial reference and learning how actions change that representation. We introduce InsightMap, a framework that uses top-down maps as both explicit spatial memory and action-conditioned prediction targets. Historical views are linked to labeled map locations, and a shared multimodal backbone jointly learns navigation action prediction and post-action map generation. Map prediction provides auxiliary training supervision, while navigation inference decodes actions from the observed spatial context. An aligned RGB-D data pipeline supports a common interface for navigation, visual question answering, situated reasoning, and 3D grounding. On the validation-unseen splits of R2R-CE and RxR-CE, InsightMap achieves success rates (SR) of 56.9% and 54.9%, respectively. Adding map-prediction supervision improves R2R-CE SR by 4.3 and success weighted by path length (SPL) by 3.2 percentage points. On static spatial tasks, InsightMap achieves 103.7 CIDEr on ScanQA, 60.1% exact-match accuracy on SQA3D, and 53.1% grounding accuracy at 0.5 IoU on ScanRefer with detected object proposals. On Unitree Go2, it outperforms NaVid and NaVILA in hallway, lab, and office environments.
Abstract:Language-guided navigation requires connecting partial observations to a persistent spatial reference and learning how actions change that representation. We introduce InsightMap, a framework that uses top-down maps as both explicit spatial memory and action-conditioned prediction targets. Historical views are linked to labeled map locations, and a shared multimodal backbone jointly learns navigation action prediction and post-action map generation. Map prediction provides auxiliary training supervision, while navigation inference decodes actions from the observed spatial context. An aligned RGB-D data pipeline supports a common interface for navigation, visual question answering, situated reasoning, and 3D grounding. On the validation-unseen splits of R2R-CE and RxR-CE, InsightMap achieves success rates (SR) of 56.9% and 54.9%, respectively. Adding map-prediction supervision improves R2R-CE SR by 4.3 and success weighted by path length (SPL) by 3.2 percentage points. On static spatial tasks, InsightMap achieves 103.7 CIDEr on ScanQA, 60.1% exact-match accuracy on SQA3D, and 53.1% grounding accuracy at 0.5 IoU on ScanRefer with detected object proposals. On Unitree Go2, it outperforms NaVid and NaVILA in hallway, lab, and office environments.
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
arXiv:2609.37187 [cs.CV]
(or arXiv:2609.37187v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.37187 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 271 cuvinte
Subiecte conexe
Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
arXiv · 12 oct. 2026
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
arXiv · 20 iul. 2026
CG-World: A Large-Scale World-State Dataset and Protocol for World Models
arXiv · 31 iul. 2026
Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models
arXiv · 24 aug. 2026