aiqbench analysis
This paper proposes GGPO, an off-policy reinforcement learning method for training language models on verifiable rewards that avoids the high-variance importance weights used by methods like GRPO, which matters for long responses. The theoretical claims rest on a KL-regularized optimality-gap derivation and finite-data bounds, while the empirical results come from math reasoning benchmarks, and the reported gains are mostly relative to specific baselines. Readers should note that the ablation shows the reference-refresh schedule carries much of the performance, so the method's benefits depend on that design choice.
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
arXiv:2610.20814v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of-$K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch. This refresh is empirically important: in an ablation, fixed-reference GGPO scores $32.96$ on the five-benchmark problem-weighted average, versus $46.58$ with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.
Abstract:Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of-$K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch. This refresh is empirically important: in an ablation, fixed-reference GGPO scores $32.96$ on the five-benchmark problem-weighted average, versus $46.58$ with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.
51 pages. Code: this https URL
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2610.20814 [cs.LG]
(or arXiv:2610.20814v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20814 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words
Related coverage
Mastering Agentic Techniques: AI Agent Reinforcement Learning
NVIDIA · Jul 1, 2026
Mastering Agentic Techniques: AI Agent Customization
NVIDIA · May 20, 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
Granite 4.2 LLMs: How They're Built
Hugging Face · Aug 25, 2026