aiqbench analysis
WALA advances robot learning by leveraging unlabelled human videos to teach models how to distinguish meaningful object interactions from static backgrounds. This approach significantly improves data efficiency and zero-shot transfer capabilities for robotic manipulation tasks without requiring extensive action annotations. The framework bridges the gap between visual understanding and executable policies, offering a scalable path toward more robust autonomous agents.
WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
arXiv:2607.11397v2 Announce Type: replace Abstract: Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.
Abstract:Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.
Robotics (cs.RO)
arXiv:2607.11397 [cs.RO]
(or arXiv:2607.11397v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2607.11397 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 289 words
Related coverage
CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations
arXiv · Jul 31, 2026
Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning
arXiv · Oct 6, 2026
GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos
arXiv · Sep 16, 2026
CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
arXiv · Jun 16, 2026