aiqbench analysis
The paper proposes a pipeline that turns human videos of manipulation into robot action supervision, using object-centric hand-object optimization and retargeting so that the resulting trajectors are physically more consistent. The reported gains on DexYCB and HO3D-v3 and on three real-robot dexterous tasks are promising, but the evidence covers only a few tasks and benchmarks. For ML readers, the key question is whether reconstructed human data can reliably substitute for scarce robot data at scale.
Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics
arXiv:2610.15769v1 Announce Type: new Abstract: Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision. Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions. To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training. Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting. For object tracking, we combine observation confidence with temporal motion constraints to select a globally consistent object pose trajectory from per-frame pose candidates. For hand refinement, we use object geometry and motion as a reference and combine staged 2D alignment with object-local contact constraints to refine hand trajectories and finger joint configurations. Experiments on DexYCB and HO3D-v3 show improved object pose accuracy and hand-object interaction quality. Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.
Abstract:Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision. Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions. To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training. Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting. For object tracking, we combine observation confidence with temporal motion constraints to select a globally consistent object pose trajectory from per-frame pose candidates. For hand refinement, we use object geometry and motion as a reference and combine staged 2D alignment with object-local contact constraints to refine hand trajectories and finger joint configurations. Experiments on DexYCB and HO3D-v3 show improved object pose accuracy and hand-object interaction quality. Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.
21 pages, including appendix. Project page: this https URL
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2610.15769 [cs.RO]
(or arXiv:2610.15769v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2610.15769 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization
arXiv · Sep 22, 2026
EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization
arXiv · Oct 14, 2026
Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
arXiv · Aug 17, 2026
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
arXiv · Jun 9, 2026