aiqbench analysis
This paper investigates failure modes in Group Policy Optimization where standard KL regularization hinders performance, proposing ZCPO to calibrate reward coefficients using relative drift. By addressing issues like residual updates and sampling noise, the method offers a more robust approach for training group-based language models without relying on traditional reference-policy constraints.
When KL Regularization Misfires in Group Policy Optimization
arXiv:2610.12161v1 Announce Type: cross Abstract: Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.
Abstract:Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2610.12161 [cs.LG]
(or arXiv:2610.12161v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.12161 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 215 words
Related coverage
Scaling RL with verl on AMD Instinct MI355X: Async Walkthrough and Sync Benchmark
AMD · Aug 18, 2026
Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction
arXiv · Oct 1, 2026
On KL-Regularized Policy Optimization
arXiv · Oct 8, 2026
The Entangled Dimensions of Decision Theory
LessWrong · Jul 30, 2026