aiqbench analysis
This arXiv paper argues that unlearning methods for large language models are often judged only on the benchmark distribution they were tuned for, so they can appear to remove information that remains recoverable. Its cross-lingual case study shows that forgetting can vary sharply when the same fact is queried in another language or script. For readers building or auditing privacy-motivated unlearning, the takeaway is that evaluations need to test access-path generalization and adversarial recovery, not just in-distribution forgetting.
Behavioral Forgetting Is Not Privacy in Machine Unlearning
arXiv:2610.21799v1 Announce Type: new Abstract: Machine unlearning aims to remove targeted information from large language models, but existing evaluations often measure forgetting only on a designated benchmark distribution. We argue that such behavioral forgetting is an insufficient standard for privacy-oriented unlearning, and that evaluation should instead test whether forgetting persists across the access paths through which the same information may be recovered. We demonstrate this with a controlled case study of cross-lingual querying: across multiple multilingual model families, forgetting varies substantially when semantically equivalent information is queried in different languages and scripts. A single access path is therefore enough to expose a real gap between behavioral forgetting and information inaccessibility, motivating broader evaluation of access-path generalization and adversarial recoverability alongside in-distribution forgetting.
Abstract:Machine unlearning aims to remove targeted information from large language models, but existing evaluations often measure forgetting only on a designated benchmark distribution. We argue that such behavioral forgetting is an insufficient standard for privacy-oriented unlearning, and that evaluation should instead test whether forgetting persists across the access paths through which the same information may be recovered. We demonstrate this with a controlled case study of cross-lingual querying: across multiple multilingual model families, forgetting varies substantially when semantically equivalent information is queried in different languages and scripts. A single access path is therefore enough to expose a real gap between behavioral forgetting and information inaccessibility, motivating broader evaluation of access-path generalization and adversarial recoverability alongside in-distribution forgetting.
Computation and Language (cs.CL)
arXiv:2610.21799 [cs.CL]
(or arXiv:2610.21799v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.21799 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 212 words
Related coverage
Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning
arXiv · Sep 2, 2026
New framework for auditing machine unlearning
Google Research · Jun 10, 2026
LLM Unlearning Evaluation with TRIAGE
arXiv · Sep 30, 2026
When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data
arXiv · Sep 9, 2026