analiza aiqbench
GVPO addresses critical training instability in Large Language Model post-training by integrating analytical solutions for KL-constrained reward maximization directly into gradient weights. This approach guarantees a unique optimal solution while supporting flexible sampling distributions, effectively overcoming limitations inherent to previous on-policy and importance sampling methods. The technique offers a theoretically grounded yet practically adaptable framework for refining model alignment with human preferences.
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
arXiv:2504.19599v4 Announce Type: replace-cross Abstract: Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
Abstract:Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.
Accepted by NeurIPS 2025. Extended version is available as "GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation" (arxiv.org/abs/2609.21432)
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2504.19599 [cs.AI]
(or arXiv:2504.19599v4 [cs.AI] for this version)
Text integral sub licența deschisă a publicației · 195 cuvinte
Subiecte conexe
GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
arXiv · 22 sept. 2026
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
Apple · 16 sept. 2026
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
arXiv · 5 iun. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026