analiza aiqbench
Future Policy Approximation (FPA) addresses the instability inherent in offline reinforcement learning for LLMs by mitigating gradient entanglement during long-horizon reasoning tasks. By weighting gradients based on a future policy estimate, this method stabilizes training and achieves performance comparable to state-of-the-art online approaches while significantly reducing computational costs. This advancement offers a practical pathway for deploying efficient post-training strategies that maintain high accuracy in complex mathematical and code-generation benchmarks without the overhead of real-time interaction.
Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
arXiv:2509.19893v3 Announce Type: replace Abstract: Reinforcement learning (RL) has emerged as a key driver of post-training for complex reasoning in large language models (LLMs), yet online RL introduces substantial instability and computational overhead. Offline RL offers a compelling alternative by decoupling generation from training; however, offline algorithms for reasoning remain under-optimized relative to their online counterparts. We revisit the potential of policy-gradient-style offline RL and address a central challenge in offline learning: gradient entanglement. In long-horizon reasoning trajectories, correct and incorrect solutions share substantial token overlap, causing gradient updates from incorrect trajectories to suppress tokens that are also critical for correct ones. We propose Future Policy Approximation (FPA), a simple offline policy-gradient method that weights gradients using an estimate of the future policy rather than the current policy, enabling proactive gradient reweighting. We estimate the future policy through logit- space extrapolation. Across three models, seven mathematical reasoning benchmarks, and three code-generation benchmarks, FPA consistently improves over strong offline baselines, including DPO, RPO, KTO, and vanilla offline RL. FPA stabilizes long-horizon training, where vanilla objectives degrade, and achieves accuracy comparable to state-of-the-art RLVR methods such as GRPO and DAPO at a fraction of the GPU hours.
Abstract:Reinforcement learning (RL) has emerged as a key driver of post-training for complex reasoning in large language models (LLMs), yet online RL introduces substantial instability and computational overhead. Offline RL offers a compelling alternative by decoupling generation from training; however, offline algorithms for reasoning remain under-optimized relative to their online counterparts. We revisit the potential of policy-gradient-style offline RL and address a central challenge in offline learning: gradient entanglement. In long-horizon reasoning trajectories, correct and incorrect solutions share substantial token overlap, causing gradient updates from incorrect trajectories to suppress tokens that are also critical for correct ones. We propose Future Policy Approximation (FPA), a simple offline policy-gradient method that weights gradients using an estimate of the future policy rather than the current policy, enabling proactive gradient reweighting. We estimate the future policy through logit- space extrapolation. Across three models, seven mathematical reasoning benchmarks, and three code-generation benchmarks, FPA consistently improves over strong offline baselines, including DPO, RPO, KTO, and vanilla offline RL. FPA stabilizes long-horizon training, where vanilla objectives degrade, and achieves accuracy comparable to state-of-the-art RLVR methods such as GRPO and DAPO at a fraction of the GPU hours.
Computation and Language (cs.CL)
arXiv:2509.19893 [cs.CL]
(or arXiv:2509.19893v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2509.19893 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 283 cuvinte
Subiecte conexe
The Alignment Problem: Computer Science Meets Analytic Philosophy
LessWrong · 18 oct. 2026
Enabling Language-specific Reasoning in Multilingual Models with Reinforcement Learning
AMD · 24 iul. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior
LessWrong · 15 oct. 2026