aiqbench analysis
This paper is squarely about AI: it studies reinforcement learning with verifiable rewards for training language models on hard math problems. Its core contribution is a diagnostic finding that negative rollouts with higher early-stage entropy predict eventual success, and a method (NAPO, built on GRPO) that uses this signal to improve learning on prompts that otherwise stall. The reported gains (Pass@256 up 10.0 points on AIME24 and 6.6 on AIME2025) are notable but come from a specific training setup under fixed rollout budgets, so they should be validated across models and benchmarks before being treated as general.
What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR
arXiv:2611.02747v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.
Title:What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2611.02747 [cs.LG]
(or arXiv:2611.02747v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.02747 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.18653/2026.emnlp-main.1821 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words