aiqbench analysis
PatchWAM demonstrates that generative visual backbones can inherently support continuous control by treating actions as compatible patches, eliminating the need for separate action heads or experts. This architectural efficiency allows a single model to simultaneously predict future scene dynamics and robot movements, achieving high success rates on standard robotics benchmarks without adding new capacity constraints. The approach underscores that interface compatibility, rather than raw model size, is the primary bottleneck when extending vision models to physical control tasks.
An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
arXiv:2609.25961v1 Announce Type: new Abstract: Generative visual models offer a foundation for learning representations of physical dynamics, yet their extension to continuous control raises a fundamental question: do visual prediction and action generation require separate computational pathways? Existing approaches usually introduce trainable action heads or separate action experts to bridge low-dimensional states and high-dimensional visual representations. In this work, we explore whether the visual backbone's existing capacity can also support control when actions are expressed in a compatible representation. Thus, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed mapping called Action-as-Patch. This allows a single model to predict both how the robot should move and what the scene may look like afterward. Visual prediction and action generation become parts of the same generative process, without a dedicated action head or separate action expert. Experiments with subsampled training windows show gains over a matched dual-expert control, while benchmark evaluations reach 91.8% success rate on LIBERO-Plus and 96.12% on RoboTwin 2.0 in a full-data setting with additional augmented demonstrations. More broadly, the result suggests that capability need not be added where it can be inherited: the constraint on extending a generative backbone is the interface a new signal is written in, not the capacity to model it.
Abstract:Generative visual models offer a foundation for learning representations of physical dynamics, yet their extension to continuous control raises a fundamental question: do visual prediction and action generation require separate computational pathways? Existing approaches usually introduce trainable action heads or separate action experts to bridge low-dimensional states and high-dimensional visual representations. In this work, we explore whether the visual backbone's existing capacity can also support control when actions are expressed in a compatible representation. Thus, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed mapping called Action-as-Patch. This allows a single model to predict both how the robot should move and what the scene may look like afterward. Visual prediction and action generation become parts of the same generative process, without a dedicated action head or separate action expert. Experiments with subsampled training windows show gains over a matched dual-expert control, while benchmark evaluations reach 91.8% success rate on LIBERO-Plus and 96.12% on RoboTwin 2.0 in a full-data setting with additional augmented demonstrations. More broadly, the result suggests that capability need not be added where it can be inherited: the constraint on extending a generative backbone is the interface a new signal is written in, not the capacity to model it.
Robotics (cs.RO)
arXiv:2609.25961 [cs.RO]
(or arXiv:2609.25961v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.25961 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 303 words
Related coverage
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026
2026.9: There's room on this bus
Home Assistant · Sep 2, 2026
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
LessWrong · Oct 5, 2026
Introducing Unit 42 Continuous Frontier AI Defense
Palo Alto Networks · Sep 22, 2026