aiqbench analysis
This paper introduces a novel two-stage framework for embodied manipulation that learns complementary predictive representations from multi-view video clips to capture both current scene geometry and its temporal evolution under interaction. By leveraging a frozen 3D foundation model during pretraining and optimizing geometric features to preserve structural dynamics, the approach significantly improves downstream policies on benchmarks like LIBERO and WidowX without requiring future observations at inference time. The work represents a meaningful advancement in robotics by effectively integrating predictive modeling directly into action generation pathways for complex bimanual and single-arm tasks.
From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation
arXiv:2610.28164v1 Announce Type: new Abstract: Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction. The code will be available at https://github.com/xuxiaoxxxx/CPR.
Abstract:Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction. The code will be available at this https URL.
Robotics (cs.RO)
arXiv:2610.28164 [cs.RO]
(or arXiv:2610.28164v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2610.28164 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 300 words
Related coverage
When do quantum views help? Complementary classical-quantum representations in multi-view fusion
arXiv · Oct 17, 2026
Complementary Supervised and Self-Supervised Representations for Out-of-Distribution Graph Learning
arXiv · Oct 7, 2026
FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
arXiv · Sep 24, 2026
Twice Upon a Time: Complementary Representations for Time-Series Reasoning
arXiv · Oct 17, 2026