Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

Abstract:Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for post-training large language models, yet the design of its baselines and learning-rate schedules remains largely heuristic. This limits our understanding of the statistical properties of policy-gradient estimators and their interaction with optimization dynamics. In this work, we develop a theoretical framework for variance-aware baseline design and adaptive learning-rate selection in RLVR. Under a KL-regularized policy-optimization setting, we establish the unbiasedness of the resulting gradient estimator, derive exact variance expressions including the KL cross-covariance, and obtain an optimization-loss upper bound that enables principled reasoning about learning dynamics. Building on these results, we prove convergence guarantees and derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of the policy gradient. We further show that the variance-optimal baseline is a gradient-weighted estimator of the KL-regularized reward, providing a principled alternative to commonly used reward-based baselines. These results lead to two complementary improvements: a variance-optimal baseline and an SNR-adaptive learning-rate rule. Experiments on Qwen3-4B-Base show that each component independently improves policy-optimization performance. The learning-rate rule can also be naturally integrated with existing policy optimization methods to yield further gains, while combining it with the variance-optimal baseline gives the full Optimal Baseline and Learning-Rate Policy Optimization (OBLR-PO) method and achieves the strongest overall performance.

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2511.23310 [stat.ML]

(or arXiv:2511.23310v3 [stat.ML] for this version)