aiqbench analysis
TRACE addresses a critical gap in Mixture-of-Experts unlearning by detecting routing mismatches between forget and retain data to prevent under-regularization of specific experts. This method achieves a 9% relative utility improvement over baselines while maintaining comparable forgetting quality, offering a more efficient path for managing model lifecycle in sparse architectures. The approach provides a practical solution for practitioners needing to balance data privacy requirements with model performance in MoE systems.
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
arXiv:2606.10338v1 Announce Type: new Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored. Unlike dense models, MoE architectures employ a router at each layer to assign each token to a sparse subset of experts. In this work, we observe that forget data often activates a small subset of experts disproportionately, while these experts may receive much weaker activation from retain data. This forget--retain routing mismatch can leave forget-critical experts under-regularized during unlearning. To address this, we propose \textbf{TRACE}, Targeted Routing-Aware Calibration of Experts, for MoE unlearning. TRACE first detects forget-critical experts from offline activation statistics, and then calibrates retain regularization by reweighting token-level retain losses so that each selected expert's retain-side activation frequency better matches its forget-side counterpart. Experiments on WMDP and MUSE-BOOKS across multiple MoE LLMs show that TRACE consistently improves the forget-utility trade-off, yielding a 9\% relative utility improvement over the strongest baseline under comparable forgetting quality and the best performance on three out of four MUSE-BOOKS metrics.
Abstract:Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored. Unlike dense models, MoE architectures employ a router at each layer to assign each token to a sparse subset of experts. In this work, we observe that forget data often activates a small subset of experts disproportionately, while these experts may receive much weaker activation from retain data. This forget--retain routing mismatch can leave forget-critical experts under-regularized during unlearning. To address this, we propose \textbf{TRACE}, Targeted Routing-Aware Calibration of Experts, for MoE unlearning. TRACE first detects forget-critical experts from offline activation statistics, and then calibrates retain regularization by reweighting token-level retain losses so that each selected expert's retain-side activation frequency better matches its forget-side counterpart. Experiments on WMDP and MUSE-BOOKS across multiple MoE LLMs show that TRACE consistently improves the forget-utility trade-off, yielding a 9\% relative utility improvement over the strongest baseline under comparable forgetting quality and the best performance on three out of four MUSE-BOOKS metrics.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.10338 [cs.CL]
(or arXiv:2606.10338v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10338 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
arXiv · Sep 24, 2026
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
arXiv · Jun 10, 2026
When Model Merging Breaks Routing: Training-Free Calibration for MoE
arXiv · Jun 3, 2026
AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE
arXiv · Jun 5, 2026