Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models

VLA Vision-Language-Action model: a robot policy that starts from a pretrained VLM backbone and adapts it to generate actions from visual observations and language instructions. Large-scale VLM pretraining is a core part of the recipe. See Pi-0 and GR00T N1.

WAM World-Action Model: a policy that starts from a pretrained world-model or video backbone and adapts it to represent or predict how the scene changes over time and emit corresponding actions. We use WAM as the term throughout this post.

VLM Vision-Language Model: a model pretrained on image-text or video-text data to produce language outputs grounded in visual inputs, usually before being adapted for robot control.

Video backbone A pretrained video model reused as the central representation or generator inside a robot policy.

World model A model that predicts a future world state, conditioned on some action abstraction such as language, robot actions, or latent actions. The predicted state may be represented as images, video, point tracks, object states, or latent features. See the classic World Models paper and NVIDIA's Cosmos world foundation model paper.

Grounding Connecting symbols (e.g. words in a language instruction) to the perceptual and motor referents that satisfy them. Language-to-action grounding in particular means turning an instruction like "pick up the red mug" into the visual percepts and motor commands that actually accomplish it. The grounding gap is the persistent shortfall between what a model knows about language and what it can reliably cause to happen in the physical world.

Inverse dynamics Given a current observation ot and a future observation ot+k, infer the most plausible action or action sequence that would produce the transition.

Joint prediction Given ot and language lt, train one policy π(ot, lt) to predict both future observations ot+1:t+k and actions at:t+k.

Action chunk A short horizon action sequence at:t+k — i.e. the k actions at, at+1, …, at+k−1 — such as joint commands, end-effector deltas, and gripper states, predicted in one policy call. See ACT and Diffusion Policy.

Mixture-of-Transformers (MoT) Several modality-specific transformers or experts, such as a video transformer and an action transformer, connected through shared attention while keeping separate weights. See the related Transfusion paper.

Diffusion Transformer (DiT) A transformer backbone used inside diffusion or flow-matching models to denoise image, video, or action tokens over multiple steps. DiT commonly uses adaptive layer normalization (adaLN) to inject timestep conditioning into transformer blocks. See the Peebles and Xie DiT paper.

VAE Variational Autoencoder: in this post, mainly image and video VAEs that compress high-resolution images or videos into latent representations before generation or policy learning. This reduces token count substantially; for example, Wan 2.1's VAE uses 4× temporal and 8×8 spatial compression, while Wan 2.2-5B uses a higher-compression 4× temporal and 16×16 spatial interface. See the original VAE paper, Rombach et al.'s latent diffusion paper, the Wan paper, and the Wan 2.2 release.

Wan A family of large pretrained video-generation models often used as the video backbone in recent WAMs. See the Wan paper.

Cosmos NVIDIA's world foundation model family for physical AI, including video prediction models that can be adapted for robotics and policy learning. See the Cosmos paper.

DROID Distributed Robot Interaction Dataset: a large real-world manipulation dataset with more than 50k demonstrations across varied tasks, collected using Franka Panda robot arms. See the DROID paper.

RoboArena A distributed real-world benchmark for evaluating generalist robot policies on open-ended language-conditioned tasks. See the RoboArena paper.

RoboLab A high-fidelity simulation benchmark for analyzing task-generalist robot policies across visual, relational, and procedural competencies. See the RoboLab paper.

CALVIN A language-conditioned manipulation benchmark focused on long-horizon task sequences in simulation. See the CALVIN paper.

LIBERO A robot-learning benchmark for studying knowledge transfer, lifelong learning, and generalization in manipulation. See the LIBERO paper.

RoboTwin A simulation data generator and benchmark for robust bimanual robotic manipulation under domain randomization. See the RoboTwin 2.0 paper.

FAST / BEAST Discrete action-tokenization methods that turn continuous robot actions into token sequences, making action learning more compatible with VLM-style training. See the FAST paper and BEAST paper.

VPP Video Prediction Policy: a WAM-style method that uses predictive visual representations from a video model to condition robot actions. See the VPP paper.

LAPA Latent Action Pretraining from Videos: a method for learning action-like latent variables from videos without ground-truth robot action labels. See the LAPA paper.

OOD Out-of-distribution: a task, object, environment, or instruction outside the examples used during training or demonstration.

FLOP / ZFLOP Floating-point operations measure training compute. 1 ZFLOP equals 1021 FLOPs.

H100 / GPU-hour H100 is a high-end NVIDIA training GPU. A GPU-hour means one GPU running for one hour, a rough unit for comparing training cost.

BF16 Brain floating point 16-bit: a lower-precision number format commonly used to train large neural networks efficiently.

I2V Image-to-video: a video-generation setup conditioned on an initial image or frame.

Background: two building blocks. A visuomotor policy maps current observations plus a goal or instruction to robot actions. A world model predicts future visual or latent states from the current state plus an action or goal abstraction. A WAM sits at the overlap: it leverages a pretrained video/world-model backbone as a prior and predicts both future states and robot actions.Visuomotor policy: language instruction and current observation in, action sequence out.World model: current world state plus an action abstraction in, future image or latent out.

Last year, my Scholar Inbox digest was dominated almost every day by new VLA papers. This changed in the last months, and a different keyword is coming up almost daily now too: WAM, short for World-Action Model. In October 2025, I wrote in my State of VLA post that WAMs were a small subfield within VLA research and far less popular than VLAs initialized from VLMs [60]. That has changed fast, and my wish to see more work in this direction has already become reality.

So what changed, and why now? Maybe it is because WAMs are the shiny new thing everyone wants to work on, or VLA authors ran out of new names for their own VLAs, since basically all "-VLA" names like "X-VLA" and "Ego-VLA" are already used. So now we can recycle them for the WAM area. But more likely it has something to do with VLM-based VLAs getting stuck. Modern VLAs benefited from massive vision-language pretraining, but they still hit a language-to-action grounding wall. The problem of mapping language and pixels into behavior still has to be learned from robot data. WAMs offer a different starting point. They use pretrained video or world-model backbones that already model how scene dynamics change under language conditioning. If that prior transfers to behavior generation, the remaining video-to-action gap may be smaller than learning language-to-action grounding directly.

But the ideas behind WAMs are not new. Early WAMs like UniPi [10] proposed essentially this approach back in 2023. So why did it take several years for the paradigm to enter the robot foundation model mainstream, and where does it actually stand today? This post takes a closer look at the modern WAM landscape to answer the central question:

Central question: Is this a real paradigm shift in research and industry, or just a short hype cycle? And if the recipe works so well, why did it take several years after early papers like UniPi for WAMs to become so