analiza aiqbench
This study provides a fine-grained decomposition of language model memorization dynamics, revealing that both recitation and recollection rely on sequence-level gradient alignment while identifying lower layers as primary drivers of this phenomenon. By demonstrating how misalignment causes forgetting in duplicated sequences and showing that intervening on specific influential parameters can ablate memorization, the work offers actionable insights for controlling training distributions without relying solely on cross-entropy baselines. These findings advance the theoretical understanding of how models fit training data tails and suggest new pathways for mitigating privacy risks through targeted parameter intervention.
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
arXiv:2609.34933v1 Announce Type: new Abstract: Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.
Title:Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Abstract:Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.34933 [cs.LG]
(or arXiv:2609.34933v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.34933 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 282 cuvinte
