aiqbench analysis
This research introduces POPO, a novel reinforcement learning framework that enhances large language model reasoning by optimizing exclusively on positive rollouts while using self-distillation to implicitly capture negative gradients. By eliminating disjoint negative samples, the method stabilizes policy updates and achieves superior performance on complex mathematical benchmarks like AIME 2025 compared to existing GRPO techniques. The approach represents a significant step forward in efficient post-training strategies for improving model accuracy without relying on sparse binary reward penalties.
Improving Reasoning Ability via Asynchronous On-Policy Self-Distillation under Positive Rollouts
arXiv:2605.06650v2 Announce Type: replace Abstract: Distillation and reinforcement learning through verifiable rewards (RLVR) have achieved progress in enhancing the reasoning ability of large language models (LLMs). However, we note that negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards. In this work, we propose Positive-Only Policy Optimization (POPO), an on-policy self-distillation integrated RLVR framework in which learning occurs exclusively on online positive rollouts. Specifically, POPO utilizes bounded importance sampling over the positive rollout set. Thus, no disjoint negative rollouts are used for gradient guidance during post-training. We show that implicit negative gradients can emerge naturally through reinforcing the positive probability via rollout redistribution. Next, POPO stabilizes the policy optimization through self-distillation. First, it applies a Siamese policy network with a momentum-based adaptation law for asynchronous policy evolution. Second, we replace the KL-divergence with a bounded similarity penalty term in the Siamese representation space. We conduct extensive experiments using publicly available, well-established text-LLM models across all-level mathematical benchmarks (MATH-500, AMC23, AIME 2024/2025, and Olympiad). Our experiment demonstrates that POPO achieves superior performance compared to GRPO. Notably, we show that POPO can achieve 36.67% in AIME 2025 with Qwen-Math-7B, outperforming GRPO 30.00%. Our ablation and sweep studies further illustrate the necessity and robustness.
Abstract:Distillation and reinforcement learning through verifiable rewards (RLVR) have achieved progress in enhancing the reasoning ability of large language models (LLMs). However, we note that negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards. In this work, we propose Positive-Only Policy Optimization (POPO), an on-policy self-distillation integrated RLVR framework in which learning occurs exclusively on online positive rollouts. Specifically, POPO utilizes bounded importance sampling over the positive rollout set. Thus, no disjoint negative rollouts are used for gradient guidance during post-training. We show that implicit negative gradients can emerge naturally through reinforcing the positive probability via rollout redistribution. Next, POPO stabilizes the policy optimization through self-distillation. First, it applies a Siamese policy network with a momentum-based adaptation law for asynchronous policy evolution. Second, we replace the KL-divergence with a bounded similarity penalty term in the Siamese representation space. We conduct extensive experiments using publicly available, well-established text-LLM models across all-level mathematical benchmarks (MATH-500, AMC23, AIME 2024/2025, and Olympiad). Our experiment demonstrates that POPO achieves superior performance compared to GRPO. Notably, we show that POPO can achieve 36.67% in AIME 2025 with Qwen-Math-7B, outperforming GRPO 30.00%. Our ablation and sweep studies further illustrate the necessity and robustness.
Computation and Language (cs.CL)
arXiv:2605.06650 [cs.CL]
(or arXiv:2605.06650v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.06650 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 310 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · Sep 24, 2026
Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI
Interconnects · Sep 22, 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · Jul 24, 2026