aiqbench analysis
This research dissects the mechanics of "spurious forgetting" in language models, revealing that apparent knowledge loss during fine-tuning often involves reversible shifts rather than permanent data deletion. By isolating shared directional changes from fact-specific updates, the study identifies specific conditions under which old memories can be recovered or where catastrophic erosion truly occurs. These insights are critical for developing more stable training protocols and understanding the true capacity of models to retain historical information alongside new facts.
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
arXiv:2610.08718v1 Announce Type: new Abstract: Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Abstract:Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.08718 [cs.CL]
(or arXiv:2610.08718v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.08718 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · Jul 21, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
arXiv · Oct 17, 2026
NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting
arXiv · Aug 3, 2026