Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Object Category-agnostic Temporal Localization

Abstract:Human videos provide a scalable source of demonstrations for robot imitation learning. However, converting them into executable and semantically aligned robot trajectories requires determining \emph{when} task-relevant interactions occur, \emph{how} demonstrated human grasps should be retargeted across embodiments, and \emph{which} post-contact motions should be reproduced. In this paper, we present \textit{HOCALT} (\emph{H}and--\emph{O}bject \emph{C}ategory--\emph{A}gnostic \emph{L}ocalization and \emph{T}ransfer), a hand-centric retargeting framework that unifies 3D hand motion reconstruction, temporal contact localization, and cross-embodiment trajectory generation. Given synchronized stereo videos, \textit{HOCALT} identifies task-relevant contact intervals by jointly reasoning over reconstructed 3D hand articulation and category-agnostic object motion cues. These intervals serve as temporal anchors for initializing cross-embodiment transfer, where the demonstrated human grasps are retargeted into multi-modal robot grasp hypotheses. Within each interval, these hypotheses are then propagated along the demonstrated hand motion, yielding executable trajectories. Finally, we augment the transferred trajectories to generate diverse variants from a single demonstration. Across various tasks, \emph{HOCALT} outperforms VLM-based temporal localization baselines, achieving higher replay success than existing retargeting approaches.

Robotics (cs.RO)

arXiv:2606.10743 [cs.RO]

(or arXiv:2606.10743v2 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2606.10743 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.