aiqbench analysis
This paper introduces Generalized Proximal Policy Optimization (GPPO), a unified framework that formalizes LLM reinforcement learning algorithms through the lens of $f$-divergence. By establishing a generalized trust-region theory, the authors demonstrate how different divergence functions and estimator topologies create distinct policy update dynamics, with existing methods like PPO emerging as special cases. Experiments on math and coding benchmarks confirm that optimizing both dimensions yields significant performance gains over standard GRPO, offering a principled approach to improving reasoning capabilities in large language models.
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
arXiv:2610.13015v1 Announce Type: new Abstract: Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.
Abstract:Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.13015 [cs.LG]
(or arXiv:2610.13015v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.13015 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
Netskope (NTSK) Q2 2027 Earnings Call Transcript
fool.com · Sep 10, 2026
Regret Optimality of Sample Average Approximation for Data-Driven Newsvendor Problems: A General Optimization Perspective
arXiv · Jul 21, 2026
Reinforcement Learning: From Algorithms To Foundation Models
arXiv · Jul 21, 2026
Weighted Low-Rank Matrix Approximation: Acceleration and Applications
arXiv · Jul 28, 2026