analiza aiqbench
This research introduces Prophet, a scalable robot world model designed to enable efficient reinforcement learning training without relying on expensive real-world interactions or hand-crafted simulators. By coupling this simulator with FlowScale for stable gradient optimization in flow-based action heads, the method achieves significant performance gains across diverse robotic backbones and environments. The work represents a critical step toward data-efficient post-training of Vision-Language-Action policies using learned dynamics rather than pure imitation.
Reinforcing Action Policies by Prophesying
arXiv:2511.20633v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.
Abstract:Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.
Robotics (cs.RO)
arXiv:2511.20633 [cs.RO]
(or arXiv:2511.20633v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2511.20633 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 257 cuvinte
Subiecte conexe
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · 24 sept. 2026
Density-Matrix Reinforcement Learning
arXiv · 14 oct. 2026
TacCoRL: Integrating Tactile Feedback into VLA via Simulation
arXiv · 12 iun. 2026
Three perspectives on corrigibility - shards, simulators, singular learning
LessWrong · 16 oct. 2026