aiqbench analysis
This research introduces VGGT-DP, a visuomotor policy framework that enhances robot generalization by integrating geometric priors from pretrained 3D perception models with proprioceptive feedback. By implementing frame-wise token reuse and random token pruning, the approach significantly reduces inference latency while improving robustness in precision-critical tasks compared to existing baselines. The work represents a meaningful advancement in leveraging vision foundation models for autonomous manipulation and spatial understanding.
VGGT-DP: Generalizable Robot Control via Vision Foundation Models
arXiv:2509.18778v2 Announce Type: replace-cross Abstract: Visual imitation learning frameworks allow robots to learn manipulation skills from expert demonstrations. While existing approaches mainly focus on policy design, they often neglect the structure and capacity of visual encoders, limiting spatial understanding and generalization. Inspired by biological vision systems, which rely on both visual and proprioceptive cues for robust control, we propose VGGT-DP, a visuomotor policy framework that integrates geometric priors from a pretrained 3D perception model with proprioceptive feedback. We adopt the Visual Geometry Grounded Transformer (VGGT) as the visual encoder and introduce a proprioception-guided visual learning strategy to align perception with internal robot states, improving spatial grounding and closed-loop control. To reduce inference latency, we design a frame-wise token reuse (FTR) mechanism that reuses cached VGGT aggregator tokens from overlapping observation frames and computes features only for the latest frame, substantially reducing redundant visual encoding. We further apply random token pruning to enhance policy robustness and reduce overfitting. Experiments on challenging MetaWorld tasks show that VGGT-DP significantly outperforms strong baselines such as DP and DP3, particularly in precision-critical and long-horizon scenarios.
Abstract:Visual imitation learning frameworks allow robots to learn manipulation skills from expert demonstrations. While existing approaches mainly focus on policy design, they often neglect the structure and capacity of visual encoders, limiting spatial understanding and generalization. Inspired by biological vision systems, which rely on both visual and proprioceptive cues for robust control, we propose VGGT-DP, a visuomotor policy framework that integrates geometric priors from a pretrained 3D perception model with proprioceptive feedback. We adopt the Visual Geometry Grounded Transformer (VGGT) as the visual encoder and introduce a proprioception-guided visual learning strategy to align perception with internal robot states, improving spatial grounding and closed-loop control. To reduce inference latency, we design a frame-wise token reuse (FTR) mechanism that reuses cached VGGT aggregator tokens from overlapping observation frames and computes features only for the latest frame, substantially reducing redundant visual encoding. We further apply random token pruning to enhance policy robustness and reduce overfitting. Experiments on challenging MetaWorld tasks show that VGGT-DP significantly outperforms strong baselines such as DP and DP3, particularly in precision-critical and long-horizon scenarios.
Robotics (cs.RO); Artificial Intelligence (cs.AI)
arXiv:2509.18778 [cs.RO]
(or arXiv:2509.18778v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2509.18778 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 269 words
Related coverage
Links #5: 2026/07
LessWrong · Jul 31, 2026
NVIDIA Research Advances Robotics From Simulation to the Real World
NVIDIA · May 28, 2026
Can Vision Foundation Models Navigate? Zero-Shot Real-World Evaluation and Lessons Learned
arXiv · Jun 17, 2026
Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation
arXiv · Sep 25, 2026