aiqbench analysis
sensVLA demonstrates how integrating spatially grounded vision with language models can enhance the fault tolerance and accuracy of autonomous heavy machinery control systems. By decoupling 3D geometry reasoning from semantic understanding, this architecture achieves superior performance in loading scenarios even when visual inputs are degraded. This work highlights a critical pathway for deploying robust AI agents in complex industrial robotics environments where safety margins are paramount.
sensVLA: Spatially-Grounded Vision-Language-Action Model for Autonomous Wheel Loader
arXiv:2609.17021v1 Announce Type: cross Abstract: Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry. We present sensVLA, a Vision-Language-Action (VLA) architecture that combines a Qwen3-2B Vision-Language Model (VLM) with a fully trainable transformer action expert trained by flow-matching velocity regression. sensVLA routes Bird's-Eye-View (BEV) features, extracted from fused front and rear lidar, directly to the action expert through a dedicated cross-attention pathway, while the VLM consumes front and rear RGB views to provide task-conditioned semantic context. This design decouples spatial grounding from linguistic reasoning while preserving interaction between both streams at decision time. The expert predicts six action dimensions: longitudinal velocity, steering, body-frame displacement, arm rate, and bucket rate. On a real-world dataset from a wheel loader, sensVLA reaches aggregate per-step parity with a strong camera-only baseline and reduces longitudinal velocity RMSE by 28% and displacement error by 9% on loading centric scenarios. It also degrades 29% less when the camera stream is corrupted or removed, evidencing that explicit spatial grounding improves accuracy and fault-tolerance for heavy equipment autonomy.
Abstract:Autonomous wheel-loader control requires joint reasoning over task semantics, egocentric vision, proprioception, and 3D scene geometry. We present sensVLA, a Vision-Language-Action (VLA) architecture that combines a Qwen3-2B Vision-Language Model (VLM) with a fully trainable transformer action expert trained by flow-matching velocity regression. sensVLA routes Bird's-Eye-View (BEV) features, extracted from fused front and rear lidar, directly to the action expert through a dedicated cross-attention pathway, while the VLM consumes front and rear RGB views to provide task-conditioned semantic context. This design decouples spatial grounding from linguistic reasoning while preserving interaction between both streams at decision time. The expert predicts six action dimensions: longitudinal velocity, steering, body-frame displacement, arm rate, and bucket rate. On a real-world dataset from a wheel loader, sensVLA reaches aggregate per-step parity with a strong camera-only baseline and reduces longitudinal velocity RMSE by 28% and displacement error by 9% on loading centric scenarios. It also degrades 29% less when the camera stream is corrupted or removed, evidencing that explicit spatial grounding improves accuracy and fault-tolerance for heavy equipment autonomy.
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
arXiv:2609.17021 [cs.CV]
(or arXiv:2609.17021v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.17021 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 270 words
Related coverage
Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super
NVIDIA · Aug 4, 2026
GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
arXiv · Oct 1, 2026
VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving
arXiv · Jun 11, 2026
Grounded Action Model: 3D Grounding as a Foundation for Robotics
arXiv · Sep 22, 2026