aiqbench analysis
This research demonstrates how fine-tuned vision-language models can solve robot localization in GPS-denied environments by treating it as a semantic reasoning task rather than pure geometric estimation. By directly regressing pose coordinates from the final hidden state of a Qwen2.5-VL-7B model, the approach achieves high accuracy while leveraging cross-modal data from cameras and LiDAR to handle occlusions or incomplete maps. The study validates that semantic spatial reasoning offers robustness comparable to traditional methods, even when relying on sparse visual cues.
Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization
arXiv:2607.01079v2 Announce Type: replace Abstract: We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task rather than a geometric estimation problem. Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-language model, given a front camera image, a polar LiDAR scan, and a top-down semantic grid map, can infer the robot pose. We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates (x, y, theta) directly from the final hidden state, bypassing text generation. Training uses a composite position-and-direction loss with curriculum learning on a custom Gazebo dataset of 120,112 samples and 527 scenes. On the in-distribution test set of 18,017 samples, the model achieves 98.23 percent position accuracy, 98.00 percent direction accuracy, 96.75 percent full pose accuracy, a mean position error of 0.11 m, and a mean orientation error of 5.7 degrees at 0.62 s per sample. Position accuracy drops by only 7.2 percentage points on seven unseen object categories, reaching 90.99 percent, supporting semantic spatial reasoning rather than appearance memorization. With incomplete maps, fine-tuning recovers performance to 93.72 percent position accuracy, showing adaptability to stale or partial map information. Two ablations highlight cross-modal complementarity. Without LiDAR, using only camera and map inputs, position accuracy remains 95.06 percent, only 3.2 percentage points below the full system. However, when the camera sees no visible objects in a wall-facing view, LiDAR sustains 92.33 percent position accuracy, compared with 70.74 percent when neither LiDAR nor visible objects are available. This shows that LiDAR becomes the primary localization signal when camera semantics are unavailable and provides a reliable fallback under occlusion or sparse layouts.
This paper has been withdrawn by Suraj Borate Mr.
Abstract:We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task rather than a geometric estimation problem. Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-language model, given a front camera image, a polar LiDAR scan, and a top-down semantic grid map, can infer the robot pose. We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates (x, y, theta) directly from the final hidden state, bypassing text generation. Training uses a composite position-and-direction loss with curriculum learning on a custom Gazebo dataset of 120,112 samples and 527 scenes. On the in-distribution test set of 18,017 samples, the model achieves 98.23 percent position accuracy, 98.00 percent direction accuracy, 96.75 percent full pose accuracy, a mean position error of 0.11 m, and a mean orientation error of 5.7 degrees at 0.62 s per sample. Position accuracy drops by only 7.2 percentage points on seven unseen object categories, reaching 90.99 percent, supporting semantic spatial reasoning rather than appearance memorization. With incomplete maps, fine-tuning recovers performance to 93.72 percent position accuracy, showing adaptability to stale or partial map information. Two ablations highlight cross-modal complementarity. Without LiDAR, using only camera and map inputs, position accuracy remains 95.06 percent, only 3.2 percentage points below the full system. However, when the camera sees no visible objects in a wall-facing view, LiDAR sustains 92.33 percent position accuracy, compared with 70.74 percent when neither LiDAR nor visible objects are available. This shows that LiDAR becomes the primary localization signal when camera semantics are unavailable and provides a reliable fallback under occlusion or sparse layouts.
Some of the results that we have submitted, we just realized we are not able to reproduce. We would like to upload later with more reliable experiments
Robotics (cs.RO)
arXiv:2607.01079 [cs.RO]
(or arXiv:2607.01079v2 [cs.RO] for this version)
From: Suraj Borate Mr. [view email] [v1] Wed, 1 Jul 2026 15:37:29 UTC (29,809 KB) [v2] Mon, 14 Sep 2026 11:10:40 UTC (1 KB) (withdrawn)
Full text under the publisher's open licence · 345 words
Related coverage
Diagnosing Visual Ignorance in Vision-Language Models
arXiv · Jun 8, 2026
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
arXiv · Oct 2, 2026
UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound
arXiv · Sep 28, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026