analiza aiqbench
Sparse Memory Finetuning (SMF) offers a viable alternative to LoRA and full finetuning by mitigating catastrophic forgetting through selective updates of key-value memory rows. While achieving a 2.5 percentage point gain on the MedMCQA task, SMF maintains base model capabilities within roughly one point of drift, outperforming standard methods that sacrifice general knowledge for task-specific gains. The study further demonstrates that row-selection strategies like KL-divergence and TF-IDF provide distinct trade-offs in balancing these forgetting metrics.
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
arXiv:2605.03229v2 Announce Type: replace-cross Abstract: Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.
Abstract:Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2605.03229 [cs.CL]
(or arXiv:2605.03229v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.03229 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 234 cuvinte