analiza aiqbench
This paper is squarely about AI: it targets reinforcement learning with verifiable rewards for training language-model reasoning and agents, and it proposes a modification to GRPO. The core contribution is a variance-based state-sampling rule (a Neyman allocation) that could reduce wasted rollout compute, but the reported gains are benchmark-specific and the "zero-cost" proxy and "outperforms on most benchmarks" claims need checking against full results and compute-matched baselines. For ML practitioners, the useful takeaway is the idea of reallocating rollouts toward informative prefixes, not a proven drop-in win.
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
arXiv:2610.15888v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $\mu_{\mathrm{var}}^\pi(s) \propto d^\pi(s)\,\sigma_\pi(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
Abstract:Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $\mu_{\mathrm{var}}^\pi(s) \propto d^\pi(s)\,\sigma_\pi(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.15888 [cs.LG]
(or arXiv:2610.15888v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.15888 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 223 cuvinte
Subiecte conexe
Why do models task game?
AI Alignment Forum · 6 aug. 2026
Custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge
AWS · 14 aug. 2026
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · 24 sept. 2026
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
arXiv · 2 oct. 2026