aiqbench analysis
3PoinTr advances robot learning by converting human videos into global 3D point-track plans, significantly reducing the need for costly physical demonstrations. The method achieves superior transferability and success rates in both simulation and real-world tasks by compressing motion cues into fixed tokens that guide policy execution. This approach effectively bridges the gap between visual observation and robotic action planning without requiring feasible human-to-robot motion replication.
3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans
arXiv:2603.08485v3 Announce Type: replace Abstract: Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: https://adamhung60.github.io/3PoinTr/
Abstract:Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: this https URL
Robotics (cs.RO)
arXiv:2603.08485 [cs.RO]
(or arXiv:2603.08485v3 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2603.08485 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words
Related coverage
3PoinTr: 3D Point Tracks for Learning Manipulation from Unconstrained Human Videos
arXiv · Jun 4, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations
arXiv · Jun 10, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026