GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

Abstract:Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where removing data inadvertently impairs model utility. While prior work has focused on architectural innovations, the influence of data-level factors remains underexplored. We introduce GUARD, a framework for **G**uided **U**nlearning **A**nd **R**etention via **D**ata Attribution. GUARD uses a lightweight proxy data attribution metric tailored to LLM unlearning, which efficiently quantifies the "alignment" between the Forget and Retain (or synthetic supplement) Sets. This metric leads to a new unlearning objective that assigns adaptive, nonuniform weights to samples, decaying exponentially with their proxy attribution scores, and thereby provably mitigates unintended retention loss. On the TOFU benchmark with a 10% forget set, applying GUARD to Gradient Ascent (GA) improves Retain Set ROUGE-L from 0.18 to 22.40 (a 124x increase) and Retain Probability from 0.06% to 17.50%, a substantial recovery from the GA collapse. Furthermore, GUARD combined with NPO (Negative Preference Optimization) outperforms Representation Misdirection for Unlearning (RMU) with 33.80% and 53.10% reductions in the Retain Set Sacrifice Rates computed from ROUGE-L and Probability ($\rho_r$ and $\rho_p$), and a 16.01% improvement in knowledge retention on the MUSE NEWS Retain Set. These findings persist when attribution scores are computed using synthetic or highly subsampled (20%) Retain Sets, where NPO+GUARD still reduces $\rho_r$ and $\rho_p$ by up to 43.89% and 48.13% over NPO. Similar improvements are observed on MUSE, with a tolerable absolute privacy leakage degradation of 13.7 compared to the best-performing existing method in the privacy category.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2506.10946 [cs.LG]

(or arXiv:2506.10946v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2506.10946 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.