EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

Abstract:Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.

Robotics (cs.RO); Artificial Intelligence (cs.AI)

arXiv:2606.08057 [cs.RO]

(or arXiv:2606.08057v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2606.08057 arXiv-issued DOI via DataCite (pending registration)