aiqbench analysis
This arXiv preprint studies how to allocate a fixed training budget in vision-language-action robot policies between action prediction and auxiliary objectives like latent-action and future-vision prediction. Its reported gains come from benchmarks (LIBERO, CALVIN) plus four real-robot tasks, so the results are promising but not yet validated at deployment scale. For ML readers, the useful takeaway is that auxiliary-target choice and sample-level supervision routing are treated as explicit design variables.
What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
arXiv:2610.21915v1 Announce Type: new Abstract: Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.
Title:What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
Abstract:Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.
26 pages, 8 figures. Haitao Shen and Hanyu Liu contributed equally
Robotics (cs.RO)
arXiv:2610.21915 [cs.RO]
(or arXiv:2610.21915v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2610.21915 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 350 words
Related coverage
From World Models to World Action Models: Rethinking Next-State Prediction
arXiv · Sep 29, 2026
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
arXiv · Sep 17, 2026
The Machines that Make the Machines
NVIDIA · Oct 7, 2026
WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
arXiv · Sep 29, 2026