Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling

Abstract:Asynchronous reinforcement-learning post-training of large language models is computationally efficient but inherently off-policy: the learner updates on rollouts generated by a stale policy, while correcting this mismatch requires importance weights that can be extremely heavy-tailed. A small number of trajectories may then dominate an update, collapsing the effective sample size (ESS) and increasing the variance and instability of the resulting gradient estimates. Existing stabilizers apply fixed interventions regardless of the realized severity of weight concentration, either discarding selected samples or reweighting them without adapting the correction strength. We introduce \emph{Target--ESS Tempered Policy Optimization} (TETPO), which replaces each importance weight $\rho$ by $\rho^\alpha$ for a data-adaptive exponent $\alpha\in[0,1]$: $\alpha=1$ preserves the original importance weights, whereas $\alpha=0$ assigns uniform coefficients. For each prompt group, TETPO selects the largest $\alpha$ whose tempered weights satisfy a target ESS fraction $\kappa$, applying little tempering to healthy groups and stronger tempering when weight degeneracy is severe. At the population level, the tempered coefficients are exact importance weights for a geometric bridge between the sampler and learner policies. The exponent is efficiently found by one-dimensional bisection, while the resulting weights satisfy deterministic bounds on normalized-weight dispersion and weighted-gradient norm that are independent of policy drift, together with a fixed-$\alpha$ bound on bridge-target discrepancy. Power tempering also introduces no explicit zero-weight mask. On mathematical-reasoning benchmarks under high staleness, TETPO outperforms strong on- and off-policy baselines.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2610.14440 [cs.LG]

(or arXiv:2610.14440v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.14440 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.