aiqbench analysis
This research addresses the critical bottleneck in robotic manipulation by introducing P2P-T, a framework that learns complex tool use directly from human video demonstrations without requiring paired robot data. By leveraging object-centric world models to extract stable pose priors, the method achieves a significant 73% performance improvement on real-world tasks while drastically reducing computational overhead. This advancement represents a major step toward scalable, general-purpose robotics powered by foundation models and human-in-the-loop learning.
From Pixel to Poses: Object-centric Tool Manipulation Learning from Human Demonstrations
arXiv:2609.35375v1 Announce Type: new Abstract: Scaling up robotic manipulation is primarily bottlenecked by the scarcity of real-world robot data. While recent approaches leverage human video demonstrations to mitigate this shortage, they remain computationally expensive and still rely on paired human-robot data for domain alignment. Although current state-of-the-arts excel at long-horizon tasks, they struggle with the delicate and precise control required for complex tool manipulation. To overcome these limitations, we introduce P2P-T, from Pixel to Poses for Tool Manipulation, a data-efficient, object-centric framework that learns tool use directly from human demonstrations. P2P-T bridges the cognitive and physical execution gap through a two-stage approach. First, pretraining an object-centric world model to extract stable pose priors; second, integrating these priors into an efficient, pose-aware low-level policy. By utilizing a robust automated data processing pipeline powered by modern foundation models, P2P-T completely bypasses the need for human-robot aligned data. This reduces overall training overhead drastically. With minimal per-task fine-tuning, our framework achieves a 73% improvement over the previous state of the art in execution performance on complex, real-world tool manipulation tasks that currently remain out of reach for standard large-scale pretrained models.
Abstract:Scaling up robotic manipulation is primarily bottlenecked by the scarcity of real-world robot data. While recent approaches leverage human video demonstrations to mitigate this shortage, they remain computationally expensive and still rely on paired human-robot data for domain alignment. Although current state-of-the-arts excel at long-horizon tasks, they struggle with the delicate and precise control required for complex tool manipulation. To overcome these limitations, we introduce P2P-T, from Pixel to Poses for Tool Manipulation, a data-efficient, object-centric framework that learns tool use directly from human demonstrations. P2P-T bridges the cognitive and physical execution gap through a two-stage approach. First, pretraining an object-centric world model to extract stable pose priors; second, integrating these priors into an efficient, pose-aware low-level policy. By utilizing a robust automated data processing pipeline powered by modern foundation models, P2P-T completely bypasses the need for human-robot aligned data. This reduces overall training overhead drastically. With minimal per-task fine-tuning, our framework achieves a 73% improvement over the previous state of the art in execution performance on complex, real-world tool manipulation tasks that currently remain out of reach for standard large-scale pretrained models.
Robotics (cs.RO); Artificial Intelligence (cs.AI)
arXiv:2609.35375 [cs.RO]
(or arXiv:2609.35375v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.35375 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding
arXiv · Jul 23, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Affordance-Based Hierarchical Reinforcement Learning for Quadruped Pedipulation
arXiv · Jun 8, 2026
SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
arXiv · Oct 7, 2026