aiqbench analysis
This research identifies "forget set curation" as the critical upstream bottleneck in machine unlearning, demonstrating how naive data selection strategies either fail to suppress specific outputs or cause unintended capability degradation. The introduced CleanSlate benchmark provides a rigorous framework for evaluating verbatim suppression of copyrighted content like songs and books while preserving model utility. These findings shift the focus from algorithmic optimization alone to the fundamental challenge of mapping high-level removal requests to precise training data subsets without collateral damage.
What to Forget in Unlearning? Forget Set Curation for Language Models
arXiv:2608.14855v1 Announce Type: new Abstract: Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch. Yet most evaluations assume that the examples to forget are already known. In realistic language-model deployments, a requester may ask a model to stop reproducing a song or book without knowing which spans, documents, quotations, or near-duplicates in a trillion-token corpus support that behavior. We study this missing upstream problem, forget set curation: mapping a suppression request to the data passed to an unlearning algorithm. We introduce CleanSlate, a benchmark for verbatim output suppression over songs and books, with model-specific extraction profiles, content-grounded QA, and capability-retention evaluations. CleanSlate exposes two failure modes. Natural lexical and exact-substring curators often yield forget sets that lead to weak suppression. An evaluation-aware curator suppresses requested continuations almost completely, but causes collateral regression on non-requested content and model-dependent capability loss. These results show that practical unlearning is not only an optimization problem once a forget set is given: the data chosen for forgetting determines both what can be unlearnt and what else is damaged.
Title:What to Forget in Unlearning? Forget Set Curation for Language Models
Abstract:Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch. Yet most evaluations assume that the examples to forget are already known. In realistic language-model deployments, a requester may ask a model to stop reproducing a song or book without knowing which spans, documents, quotations, or near-duplicates in a trillion-token corpus support that behavior. We study this missing upstream problem, forget set curation: mapping a suppression request to the data passed to an unlearning algorithm. We introduce CleanSlate, a benchmark for verbatim output suppression over songs and books, with model-specific extraction profiles, content-grounded QA, and capability-retention evaluations. CleanSlate exposes two failure modes. Natural lexical and exact-substring curators often yield forget sets that lead to weak suppression. An evaluation-aware curator suppresses requested continuations almost completely, but causes collateral regression on non-requested content and model-dependent capability loss. These results show that practical unlearning is not only an optimization problem once a forget set is given: the data chosen for forgetting determines both what can be unlearnt and what else is damaged.
Computation and Language (cs.CL)
arXiv:2608.14855 [cs.CL]
(or arXiv:2608.14855v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.14855 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 282 words
Related coverage
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
arXiv · Jun 6, 2026
Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness
arXiv · Aug 6, 2026
REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference
arXiv · Jun 8, 2026
Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance
arXiv · Jun 6, 2026