analiza aiqbench
This research introduces MAESTRO, a novel pruning technique for Mixture-of-Experts models that utilizes Markov chains to analyze cross-layer routing dependencies rather than relying on local heuristics. By preserving the global structure of expert activation trajectories, the method achieves superior performance retention and consistency across diverse tasks like safety and ethics under significant compression constraints. This advancement directly addresses critical deployment bottlenecks in MoE architectures by optimizing memory usage without sacrificing model capabilities.
It Takes a MAESTRO To Prune Bad Experts
arXiv:2607.08601v2 Announce Type: replace Abstract: Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.
Abstract:Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.
Computation and Language (cs.CL)
arXiv:2607.08601 [cs.CL]
(or arXiv:2607.08601v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.08601 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 242 cuvinte
Subiecte conexe
From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon
BAIR Blog · 29 iul. 2026
InferOpt: Constrained Multi-Objective Search for LLM Inference Configurations
arXiv · 7 oct. 2026
Introducing the Third Generation of Apple’s Foundation Models
Apple · 8 iun. 2026
Compressing Streaming Neural Audio Encoders via Latent-Space Distillation
Apple · 24 sept. 2026