aiqbench analysis
This research introduces CLAM, a framework enabling robot learning from unlabeled video demonstrations by inferring continuous latent actions through self-supervised dynamics prediction. The approach significantly boosts task success rates on high-dimensional control problems without requiring expensive expert action labels, bridging the gap between observation-only data and executable motor commands. By effectively leveraging play data to ground these latents into real-world hardware deployments, this work advances practical robot learning efficiency.
CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations
arXiv:2505.04999v2 Announce Type: replace Abstract: Learning robot control policies from demonstrations typically requires action-labeled expert data, which is expensive to collect through teleoperation. We study a more practical setting in which expert demonstrations are available only as observation sequences without action labels, and only task-agnostic play data contains actions. We introduce continuous latent action models (CLAM), a framework that infers continuous latent actions between consecutive observations using self-supervised dynamics prediction. To ground these latent actions into executable motor commands, CLAM jointly trains an action decoder using a small amount of task-agnostic play data. We show that continuous latent actions combined with this joint training are essential for high-dimensional continuous control. Across DMControl locomotion, MetaWorld manipulation, and real-world WidowX robot tasks, CLAM improves average task success rates by 2-3x over prior latent-action baselines and approaches behavior cloning trained with privileged expert action labels. Our results demonstrate that effective robot policies can be learned from unlabeled demonstrations and deployed on real hardware without collecting expert action-labeled data. Videos and code are available at clamrobot.github.io.
Abstract:Learning robot control policies from demonstrations typically requires action-labeled expert data, which is expensive to collect through teleoperation. We study a more practical setting in which expert demonstrations are available only as observation sequences without action labels, and only task-agnostic play data contains actions. We introduce continuous latent action models (CLAM), a framework that infers continuous latent actions between consecutive observations using self-supervised dynamics prediction. To ground these latent actions into executable motor commands, CLAM jointly trains an action decoder using a small amount of task-agnostic play data. We show that continuous latent actions combined with this joint training are essential for high-dimensional continuous control. Across DMControl locomotion, MetaWorld manipulation, and real-world WidowX robot tasks, CLAM improves average task success rates by 2-3x over prior latent-action baselines and approaches behavior cloning trained with privileged expert action labels. Our results demonstrate that effective robot policies can be learned from unlabeled demonstrations and deployed on real hardware without collecting expert action-labeled data. Videos and code are available at this http URL.
Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2505.04999 [cs.RO]
(or arXiv:2505.04999v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2505.04999 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
Related coverage
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026
RotVLA: Rotational Latent Action for Vision-Language-Action Model
arXiv · Sep 25, 2026
ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
arXiv · Oct 7, 2026