aiqbench analysis
This research introduces RA$^2$, a novel latent-space technique for LLM unlearning that specifically addresses "forget-adjacent utility loss," where models degrade on benign tasks near the boundaries of removed knowledge. By redirecting forget representations toward retain-supported states, the method preserves local reasoning capabilities better than standard baselines while maintaining overall utility. The work advances the technical understanding of representation migration dynamics essential for robust model sanitization and safety.
RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
arXiv:2610.19385v1 Announce Type: new Abstract: Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA$^2$ (Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA$^2$ yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at https://github.com/ZacLing/retain_anchored_attraction.
Abstract:Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA$^2$ (Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA$^2$ yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at this https URL.
Machine Learning (cs.LG)
arXiv:2610.19385 [cs.LG]
(or arXiv:2610.19385v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.19385 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 253 words
Related coverage
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
arXiv · Jun 6, 2026
Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning
arXiv · Jun 10, 2026
LLM Unlearning Evaluation with TRIAGE
arXiv · Sep 30, 2026
Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
arXiv · Aug 25, 2026