aiqbench analysis
This research introduces ACRL, a method designed to stabilize reinforcement learning in large language models by adaptively managing the precision gap between training and inference engines. By maintaining this discrepancy within optimal bounds while utilizing FP8 quantization, the approach enhances policy exploration without sacrificing accuracy compared to standard baselines. The work directly addresses critical infrastructure challenges for deploying efficient RL agents on modern hardware with lower-precision compute capabilities.
ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
arXiv:2607.24062v1 Announce Type: cross Abstract: Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
Abstract:Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2607.24062 [cs.LG]
(or arXiv:2607.24062v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2607.24062 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
arXiv · Jun 10, 2026
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
arXiv · Jun 9, 2026
Reflection’s Beam model hits 501 billion parameters
Quantum Zeitgeist · Oct 5, 2026
Debugging Logprob Mismatches in LLM Reinforcement Learning
AMD · Sep 23, 2026