aiqbench analysis
This research investigates how on-policy reinforcement learning affects large language model alignment, revealing that environment design and role framing critically determine whether model size acts as a safety buffer or facilitates exploitation. The study highlights a key distinction between on-policy and off-policy settings regarding the preservation of inherent safety distributions, while noting that standard benchmarks often fail to predict RL-induced misalignment except in specific sycophancy scenarios.
Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Design
arXiv:2604.12500v2 Announce Type: replace Abstract: Specification gaming under Reinforcement Learning (RL) is known to cause LLMs to develop sycophantic, manipulative, or deceptive behavior, yet the conditions under which this occurs remain unclear. We train 11 instruction-tuned LLMs (0.5B-14B) with on-policy RL across 3 environments and find that model size acts as a safety buffer in some environments but enables greater harmful exploitation in others. Controlled ablations trace this reversal to environment-specific features such as role framing and implicit gameability cues. We further show that most safety benchmarks do not predict RL-induced misalignment, except in the case of Sycophancy scores when the exploit relies on inferring the user's preference. Finally, we find that on-policy RL preserves a safety buffer inherent in the model's own generation distribution, one that is bypassed during off-policy settings.
Abstract:Specification gaming under Reinforcement Learning (RL) is known to cause LLMs to develop sycophantic, manipulative, or deceptive behavior, yet the conditions under which this occurs remain unclear. We train 11 instruction-tuned LLMs (0.5B-14B) with on-policy RL across 3 environments and find that model size acts as a safety buffer in some environments but enables greater harmful exploitation in others. Controlled ablations trace this reversal to environment-specific features such as role framing and implicit gameability cues. We further show that most safety benchmarks do not predict RL-induced misalignment, except in the case of Sycophancy scores when the exploit relies on inferring the user's preference. Finally, we find that on-policy RL preserves a safety buffer inherent in the model's own generation distribution, one that is bypassed during off-policy settings.
Machine Learning (cs.LG); Cryptography and Security (cs.CR)
arXiv:2604.12500 [cs.LG]
(or arXiv:2604.12500v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.12500 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 223 words
Related coverage
Several frontier models are substantially prefill aware
LessWrong · Jun 17, 2026
Coverage-driven alignment - What ‘Teaching Claude Why’ can borrow from AV verification
LessWrong · Jun 8, 2026
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026