aiqbench analysis
This research addresses the critical training-inference mismatch in reinforcement learning for large language models by introducing an additive "score centering" correction to counteract performance drift. The method proves robust across model scales from 0.6B to 30B parameters, matching or exceeding importance sampling baselines even under severe quantization constraints. By composing with existing techniques like importance sampling, this approach offers a practical path to stabilizing RL pipelines without sacrificing rollout efficiency.
Score Centering Stabilizes Off-policy Reinforcement Learning
arXiv:2609.20807v1 Announce Type: new Abstract: Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
Abstract:Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
Machine Learning (cs.LG)
arXiv:2609.20807 [cs.LG]
(or arXiv:2609.20807v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.20807 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 238 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Automated Data Engineering and Feature Selection for the Case Study of Warpage Detection in Fused Deposition Modeling
arXiv · Jul 23, 2026
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
arXiv · Jun 6, 2026
Meet Harness-1: A 20B Retrieval Subagent Trained With Reinforcement Learning Inside a Stateful Search Harness on gpt-oss-20b
MarkTechPost · Jun 7, 2026