analiza aiqbench
This research advances Vision-Language-Action models by integrating SAM3D to enhance 3D object understanding during training, enabling robots to handle occlusions and spatial variations without needing explicit depth sensors at inference time. The approach successfully internalizes fine-grained geometric priors into the policy network while maintaining an efficient RGB-language-to-action pipeline for real-world manipulation tasks.
SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models
arXiv:2607.25912v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong potential for general robot manipulation, but most existing models rely on 2D visual-language backbones and lack fine-grained 3D understanding of target objects, especially under occlusion, pose variation, scale changes, and precise spatial interaction. We propose an object-centric 3D representation alignment framework built upon $\pi_0$, using SAM3D as a frozen 3D teacher to provide target-object 3D priors during training. Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations, which are aligned with intermediate visual features of $\pi_0$. This enables the policy to internalize target-object 3D information while preserving the original RGB-language-to-action inference pipeline without requiring depth, point clouds, masks, SAM3D, or additional 3D modules at test time. Simulation experiments show consistent improvements, achieving 99.1\% on LIBERO and an average length of 4.11 on CALVIN. Real-world experiments further demonstrate that our method is particularly effective in long-horizon manipulation scenarios where the robot must focus on different target objects across multiple subtasks.
Abstract:Vision-Language-Action (VLA) models have shown strong potential for general robot manipulation, but most existing models rely on 2D visual-language backbones and lack fine-grained 3D understanding of target objects, especially under occlusion, pose variation, scale changes, and precise spatial interaction. We propose an object-centric 3D representation alignment framework built upon $\pi_0$, using SAM3D as a frozen 3D teacher to provide target-object 3D priors during training. Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations, which are aligned with intermediate visual features of $\pi_0$. This enables the policy to internalize target-object 3D information while preserving the original RGB-language-to-action inference pipeline without requiring depth, point clouds, masks, SAM3D, or additional 3D modules at test time. Simulation experiments show consistent improvements, achieving 99.1\% on LIBERO and an average length of 4.11 on CALVIN. Real-world experiments further demonstrate that our method is particularly effective in long-horizon manipulation scenarios where the robot must focus on different target objects across multiple subtasks.
Robotics (cs.RO); Artificial Intelligence (cs.AI)
arXiv:2607.25912 [cs.RO]
(or arXiv:2607.25912v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2607.25912 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 188 cuvinte
Subiecte conexe
BRICS Draws a New Global Line: Full Text of New Delhi Declaration
slguardian.org · 12 sept. 2026
SlotDiT: Object-Centric Representations for Diffusion Transformers
arXiv · 16 sept. 2026
LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment
arXiv · 29 sept. 2026
ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction
arXiv · 15 oct. 2026