aiqbench analysis
This arXiv paper argues that simple periodic parameter interpolation during fine-tuning, via a method it calls Sequential Fine-tuning with Averaging (SFA) and a replay-free variant of Lookahead, can reduce catastrophic forgetting about as well as replay buffers, auxiliary losses, and adaptive merging. The reported gains are for Qwen3-14B adapter continual learning, with claims of up to 36.3% less task training time, and the paper also reports competitive results on language and vision benchmarks. Readers should weigh these as preprint results on specific setups, and check whether the baselines were tuned as carefully as the proposed methods before treating the simpler approach as a default.
Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives
arXiv:2501.05559v2 Announce Type: replace Abstract: Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.
Abstract:Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2501.05559 [cs.LG]
(or arXiv:2501.05559v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2501.05559 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 221 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
How can LLM RL Work Despite Information-Theoretic Inefficiency
LessWrong · Oct 16, 2026
Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation
arXiv · Jun 8, 2026
Federated Continual Learning as a Distributed Drift-Plus-Penalty Control Problem
arXiv · Aug 25, 2026