analiza aiqbench
This study provides mechanistic evidence that reinforcement learning preserves internal model circuits better than supervised fine-tuning, offering a structural explanation for RL's superior resistance to catastrophic forgetting. The research reveals a clear trade-off where SFT achieves faster task adaptation at the cost of significant circuit disruption, whereas RL maintains base capabilities more robustly but adapts more slowly. For practitioners, this suggests that circuit preservation metrics could serve as a key indicator for selecting fine-tuning strategies that balance new task learning with the retention of prior knowledge.
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
arXiv:2605.28860v2 Announce Type: replace Abstract: Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.
Title:Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
Abstract:Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.28860 [cs.LG]
(or arXiv:2605.28860v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.28860 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 272 cuvinte
Subiecte conexe
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · 21 iul. 2026
A Mike's-Eye View of ARC's Research
AI Alignment Forum · 9 iun. 2026
Red-teaming LLM unlearning: LUNAR's "forgotten" knowledge is still recoverable
LessWrong · 23 iul. 2026