analiza aiqbench
This research establishes a theoretical limit where reinforcement learning with verifiable rewards (RLVR) inevitably causes language drift in chains of thought, making it impossible to improve model monitorability without sacrificing performance on novel reasoning tasks. The study proves that this degradation is specific to RLVR optimization pressure rather than supervised fine-tuning, highlighting a fundamental trade-off at the frontier of LLM development.
On Language Drift during RLVR Post-Training
arXiv:2610.02015v1 Announce Type: cross Abstract: Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.
Abstract:Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.02015 [cs.LG]
(or arXiv:2610.02015v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.02015 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 256 cuvinte
Subiecte conexe
How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift
arXiv · 28 iul. 2026
VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning
arXiv · 10 oct. 2026
LLMs are (still) mostly powered by imitative learning, not RL
LessWrong · 24 iul. 2026
Granite 4.2 LLMs: How They're Built
Hugging Face · 25 aug. 2026