aiqbench analysis
This research introduces GEM3D, a framework that learns amodal 3D representations from single-view RGB-D data to enable robust robotic manipulation without relying on multi-view inputs during inference. By combining point-cloud reconstruction with Gaussian splatting for novel-view consistency, the method successfully transfers geometric understanding into deployable visuomotor policies, demonstrating strong zero-shot generalization to unseen camera poses. The work advances the field of embodied AI by addressing critical limitations in viewpoint robustness and representation fidelity for real-world robotic tasks.
Learning Geometrically-Grounded Amodal 3D Representations for View-Generalizable Robotic Manipulation
arXiv:2601.22988v2 Announce Type: replace Abstract: Real-world robotic manipulation requires visuomotor policies capable of robust 3D scene reasoning under varying camera viewpoints. While recent 3D-aware manipulation policies have shown promise, they still face several limitations: (i) reliance on multi-view observations during inference, which is impractical in camera-constrained deployments; (ii) insufficient geometric fidelity in learned representations, limiting precise and viewpoint-robust control; and (iii) the lack of effective mechanisms for transferring pretrained 3D representations to downstream visuomotor policies. To address these challenges, we present GEM3D (Geometrically-Grounded 3D Manipulation), a unified 3D representation and policy learning framework for view-generalizable robotic manipulation. GEM3D learns amodal 3D representations from single-view RGB-D observations by jointly enforcing point-cloud reconstruction and Gaussian-splatting-based novel-view consistency during pretraining. A multi-step distillation strategy then transfers the learned geometric understanding into a deployable single-view visuomotor policy for downstream manipulation control. Experiments on 12 RLBench tasks and three real-world tasks demonstrate strong manipulation performance. Further zero-shot evaluations on eight RLBench tasks and the real-world tasks show robust generalization to unseen camera poses and substantial gains over existing methods.
Abstract:Real-world robotic manipulation requires visuomotor policies capable of robust 3D scene reasoning under varying camera viewpoints. While recent 3D-aware manipulation policies have shown promise, they still face several limitations: (i) reliance on multi-view observations during inference, which is impractical in camera-constrained deployments; (ii) insufficient geometric fidelity in learned representations, limiting precise and viewpoint-robust control; and (iii) the lack of effective mechanisms for transferring pretrained 3D representations to downstream visuomotor policies. To address these challenges, we present GEM3D (Geometrically-Grounded 3D Manipulation), a unified 3D representation and policy learning framework for view-generalizable robotic manipulation. GEM3D learns amodal 3D representations from single-view RGB-D observations by jointly enforcing point-cloud reconstruction and Gaussian-splatting-based novel-view consistency during pretraining. A multi-step distillation strategy then transfers the learned geometric understanding into a deployable single-view visuomotor policy for downstream manipulation control. Experiments on 12 RLBench tasks and three real-world tasks demonstrate strong manipulation performance. Further zero-shot evaluations on eight RLBench tasks and the real-world tasks show robust generalization to unseen camera poses and substantial gains over existing methods.
Robotics (cs.RO)
arXiv:2601.22988 [cs.RO]
(or arXiv:2601.22988v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2601.22988 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 260 words
Related coverage
3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints
arXiv · Sep 9, 2026
G$^3$VLA: Geometric Inductive Bias for Vision-Language-Action Models
arXiv · Oct 15, 2026
M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis
arXiv · Sep 25, 2026
MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation
arXiv · Sep 30, 2026