aiqbench analysis
TENP introduces a structured pruning framework for Mixture-of-Experts models that selectively removes neurons within less critical experts while preserving routing topology. By retaining parameters in a trapezoidal pattern across layers, the method achieves significant parameter reduction with minimal accuracy loss and improved code generation performance on Qwen and DeepSeek architectures. This approach addresses the deployment bottleneck of static parameter footprints without relying on unstructured weight pruning or complete expert removal.
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
arXiv:2606.09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, reserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.
Abstract:Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, reserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2606.09885 [cs.LG]
(or arXiv:2606.09885v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.09885 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 282 words
Related coverage
SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment
arXiv · Sep 3, 2026
How to Size GPUs for AI Inference and TCO Without Overspending
NVIDIA · Sep 1, 2026
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
arXiv · Jun 10, 2026
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
arXiv · Jul 21, 2026