analiza aiqbench
This research addresses the inherent ambiguity in unsupervised action learning by proposing Observed Transition Factorization (OTF) to decompose scene dynamics into reusable primitives rather than attempting direct recovery of true actions. The resulting state-conditioned latent actions demonstrate superior alignment and downstream policy performance, particularly when handling distractors or multi-object environments where traditional models struggle.
Latent Actions from Factorized Transition Effects under Agent Ambiguity
arXiv:2606.30544v2 Announce Type: replace Abstract: Latent Action Models (LAMs) learn action-like proxies from observation. However, in multi-object or distractor-rich scenes, observations contain not only agent motion but also distractors, camera dynamics, and background changes, making recovery of the underlying action intrinsically ambiguous without supervision. We argue that the appropriate unsupervised target is therefore not the true action itself, but a state-conditioned compositional summary of the transition effects present in the scene, enabling better action alignment and more effective utilization of action supervision. To this end, we propose a two-stage framework. We first pretrain Observed Transition Factorization (OTF) to discover reusable local transition primitives using a compositional codebook. We then aggregate these primitives into compact state-conditioned latent actions, instantiated as OTF-LAM-Pixel under the standard inverse-forward dynamics framework and OTF-LAM-Dino, a decoder-free variant operating in a frozen DINOv2 representation space. Experiments show that the learned transition primitives transfer across visual appearance and morphology, while the resulting latent actions exhibit substantially stronger action alignment than existing LAMs, make more effective use of downstream action supervision, and achieve competitive or superior downstream policy performance.
Abstract:Latent Action Models (LAMs) learn action-like proxies from observation. However, in multi-object or distractor-rich scenes, observations contain not only agent motion but also distractors, camera dynamics, and background changes, making recovery of the underlying action intrinsically ambiguous without supervision. We argue that the appropriate unsupervised target is therefore not the true action itself, but a state-conditioned compositional summary of the transition effects present in the scene, enabling better action alignment and more effective utilization of action supervision. To this end, we propose a two-stage framework. We first pretrain Observed Transition Factorization (OTF) to discover reusable local transition primitives using a compositional codebook. We then aggregate these primitives into compact state-conditioned latent actions, instantiated as OTF-LAM-Pixel under the standard inverse-forward dynamics framework and OTF-LAM-Dino, a decoder-free variant operating in a frozen DINOv2 representation space. Experiments show that the learned transition primitives transfer across visual appearance and morphology, while the resulting latent actions exhibit substantially stronger action alignment than existing LAMs, make more effective use of downstream action supervision, and achieve competitive or superior downstream policy performance.
Artificial Intelligence (cs.AI)
arXiv:2606.30544 [cs.AI]
(or arXiv:2606.30544v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.30544 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 267 cuvinte
Subiecte conexe
FlowMo-WM: A World Model with Object Momentum and Hidden Ambient Drift
arXiv · 16 iun. 2026
Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
LessWrong · 27 iul. 2026
Latent Action Reparameterization for Efficient Agent Inference
arXiv · 29 sept. 2026
Automating coherent long-form video generation
Google Research · 24 sept. 2026