analiza aiqbench
This research advances the efficiency of training Vision-Language-Action (VLA) policies by decoupling design choices in advantage-guided reinforcement learning. By introducing stage-specific offline evaluation methods, the study identifies a modular recipe that significantly improves task success on real-world bimanual robots without requiring exhaustive physical testing for every configuration. The findings provide a rigorous framework for optimizing post-training procedures using limited data, directly addressing critical bottlenecks in deploying embodied AI systems.
Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies
arXiv:2609.28161v1 Announce Type: new Abstract: Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
Abstract:Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
Robotics (cs.RO)
arXiv:2609.28161 [cs.RO]
(or arXiv:2609.28161v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2609.28161 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 260 cuvinte
Subiecte conexe
Links #5: 2026/07
LessWrong · 31 iul. 2026
HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interface
arXiv · 18 sept. 2026
Links #3: 2026/06 Part 1
LessWrong · 15 iun. 2026
Accelerate multimodal RL training with SkyRL on Amazon SageMaker HyperPod
AWS · 25 sept. 2026