analiza aiqbench
This research establishes a theoretical framework for Reinforcement Learning with Verifiable Rewards (RLVR), addressing the heuristic nature of current baseline and learning-rate designs. By deriving exact variance expressions and convergence guarantees, the authors introduce an SNR-adaptive schedule and a gradient-weighted optimal baseline that demonstrably improve policy optimization on large language models. These advances provide principled methods to reduce estimator noise without relying solely on empirical trial-and-error tuning.
Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
arXiv:2511.23310v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for post-training large language models, yet the design of its baselines and learning-rate schedules remains largely heuristic. This limits our understanding of the statistical properties of policy-gradient estimators and their interaction with optimization dynamics. In this work, we develop a theoretical framework for variance-aware baseline design and adaptive learning-rate selection in RLVR. Under a KL-regularized policy-optimization setting, we establish the unbiasedness of the resulting gradient estimator, derive exact variance expressions including the KL cross-covariance, and obtain an optimization-loss upper bound that enables principled reasoning about learning dynamics. Building on these results, we prove convergence guarantees and derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of the policy gradient. We further show that the variance-optimal baseline is a gradient-weighted estimator of the KL-regularized reward, providing a principled alternative to commonly used reward-based baselines. These results lead to two complementary improvements: a variance-optimal baseline and an SNR-adaptive learning-rate rule. Experiments on Qwen3-4B-Base show that each component independently improves policy-optimization performance. The learning-rate rule can also be naturally integrated with existing policy optimization methods to yield further gains, while combining it with the variance-optimal baseline gives the full Optimal Baseline and Learning-Rate Policy Optimization (OBLR-PO) method and achieves the strongest overall performance.
Abstract:Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for post-training large language models, yet the design of its baselines and learning-rate schedules remains largely heuristic. This limits our understanding of the statistical properties of policy-gradient estimators and their interaction with optimization dynamics. In this work, we develop a theoretical framework for variance-aware baseline design and adaptive learning-rate selection in RLVR. Under a KL-regularized policy-optimization setting, we establish the unbiasedness of the resulting gradient estimator, derive exact variance expressions including the KL cross-covariance, and obtain an optimization-loss upper bound that enables principled reasoning about learning dynamics. Building on these results, we prove convergence guarantees and derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of the policy gradient. We further show that the variance-optimal baseline is a gradient-weighted estimator of the KL-regularized reward, providing a principled alternative to commonly used reward-based baselines. These results lead to two complementary improvements: a variance-optimal baseline and an SNR-adaptive learning-rate rule. Experiments on Qwen3-4B-Base show that each component independently improves policy-optimization performance. The learning-rate rule can also be naturally integrated with existing policy optimization methods to yield further gains, while combining it with the variance-optimal baseline gives the full Optimal Baseline and Learning-Rate Policy Optimization (OBLR-PO) method and achieves the strongest overall performance.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2511.23310 [stat.ML]
(or arXiv:2511.23310v3 [stat.ML] for this version)
Text integral sub licența deschisă a publicației · 228 cuvinte
Subiecte conexe
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · 2 iun. 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · 24 sept. 2026
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
arXiv · 2 oct. 2026
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
arXiv · 8 iun. 2026