Learning with a Single Rollout via Pass@k Policy Optimization

arXiv:2606.25451v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) typically requires multiple rollouts per prompt to estimate relative credit, incurring substantial sampling cost and exacerbating training asynchrony. This limitation is particularly severe for scientific reasoning agents, where each trajectory may involve long-horizon reasoning, multiple turns, and parallel sub-runs. We study single-rollout reinforcement learning, where only one rollout is collected per prompt, and find that naive training can stagnate or become severely unstable. We introduce Single-Rollout Pass@k Policy Optimization (SR-PPO), which combines Monte Carlo token-level advantage estimation with Pass@k gradient reweighting. An online critic without large-scale pretraining is used both as a prefix-dependent baseline for credit assignment and to estimate the Pass@k gradient weight. Using only one rollout per prompt, SR-PPO achieves stable learning and performance comparable to GRPO across reasoning and logic benchmarks, even under off-policy drift that degrades the performance of standard Pass@1 PPO. A key factor in this stability is Pass@k reweighting, which emphasizes harder prompts while downweighting high-success prompts, for which imperfect credit estimates tend to produce noisier policy updates. Through analyses of critic quality, gradient noise, and update behavior, we investigate how Pass@k reweighting stabilizes single-rollout learning and why residual critic errors may limit SR-PPO's ability to outperform GRPO.

Abstract:Group Relative Policy Optimization (GRPO) typically requires multiple rollouts per prompt to estimate relative credit, incurring substantial sampling cost and exacerbating training asynchrony. This limitation is particularly severe for scientific reasoning agents, where each trajectory may involve long-horizon reasoning, multiple turns, and parallel sub-runs. We study single-rollout reinforcement learning, where only one rollout is collected per prompt, and find that naive training can stagnate or become severely unstable. We introduce Single-Rollout Pass@k Policy Optimization (SR-PPO), which combines Monte Carlo token-level advantage estimation with Pass@k gradient reweighting. An online critic without large-scale pretraining is used both as a prefix-dependent baseline for credit assignment and to estimate the Pass@k gradient weight. Using only one rollout per prompt, SR-PPO achieves stable learning and performance comparable to GRPO across reasoning and logic benchmarks, even under off-policy drift that degrades the performance of standard Pass@1 PPO. A key factor in this stability is Pass@k reweighting, which emphasizes harder prompts while downweighting high-success prompts, for which imperfect credit estimates tend to produce noisier policy updates. Through analyses of critic quality, gradient noise, and update behavior, we investigate how Pass@k reweighting stabilizes single-rollout learning and why residual critic errors may limit SR-PPO's ability to outperform GRPO.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2606.25451 [cs.LG]

(or arXiv:2606.25451v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.25451 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Learning with a Single Rollout via Pass@k Policy Optimization

Related coverage