aiqbench analysis
This arXiv preprint proposes a server-side method for merging client-trained unlearning adapters in federated LLM settings, using Fisher statistics to normalize the merge and protect retain-sensitive parameters. Its main claim is that this reduces the magnitude explosion that degrades utility under stronger unlearning. Results are reported on WMDP, TOFU, and MUSE-HP with Zephyr-7B and Llama-3-8B, but the gains are described as metric-dependent, so practitioners should check whether the trade-offs hold for their own forgetting targets.
FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning
arXiv:2610.21720v1 Announce Type: cross Abstract: Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.
Abstract:Federated LLM unlearning addresses targeted knowledge removal when forget and retain data remain distributed across private clients and are inaccessible to the server. The resulting locally trained adapters encode conflicting objectives, making their aggregation fundamentally different from conventional federated optimization. A key failure mode is magnitude explosion, whereby unnormalized cross-client aggregation increases the merged update magnitude and degrades retained utility under stronger unlearning. We propose FIRM (Fisher-Informed Retention Merging), a single-round framework for stable and retention-aware server-side aggregation. FIRM uses client-local Fisher statistics as a unified sensitivity signal to construct a Fisher-normalized base merge and impose explicit control over retain-sensitive parameters. Its second-order retention projection admits a unique minimum-distance solution, while the corresponding dual scales with the number of retain clients rather than the model dimension. Experiments across WMDP, TOFU, and MUSE-HP using Zephyr-7B and Llama-3-8B show utility gains at comparable forgetting on WMDP and MUSE-HP, with metric-dependent trade-offs on TOFU. Controlled comparisons support benefits of the combined base merge and additional retention correction at selected operating points.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.21720 [cs.LG]
(or arXiv:2610.21720v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.21720 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 269 words
Related coverage
Stock Market Today Highlights, Sept 2: Sensex ends 373 pts lower at 76, 570, Nifty below 24, 000; Asian Paints, HDFC Bank fall most
thehindubusinessline.com · Sep 2, 2026
Why I Left Google DeepMind
LessWrong · Jul 15, 2026
Dynamic Entanglement-Weighted Pruning for Quantum Federated Unlearning in Supply-Chain Risk Prediction
arXiv · Aug 20, 2026
A Model Merging Approach for Continual MLLM Unlearning
arXiv · Aug 6, 2026