analiza aiqbench
The paper identifies a tension in modern mixture-of-experts designs: shared experts, which are active for every token, reduce the variability of token representations, which in turn limits routing exploration. Fine-grained expert segmentation expands the routing space and needs more exploration, so the expressiveness gains may go unrealized. MirrorMoE proposes token-adaptive suppression of unnecessary experts as a remedy, with reported gains across model scales. Readers should look for independent benchmarks and ablations, since the abstract alone does not establish how the gains compare to other routing fixes.
Rethinking Routing Exploration in Modern Mixture-of-Experts
arXiv:2610.22353v1 Announce Type: cross Abstract: Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
Abstract:Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.22353 [cs.LG]
(or arXiv:2610.22353v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.22353 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 231 cuvinte
Subiecte conexe
Dropless MoE Training in JAX with Primus-Turbo
AMD · 10 iun. 2026
Digital States 2026: AI Moves From Experiment to Infrastructure
govtech.com · 24 sept. 2026
Long Context, One Node: Serving GLM-5.3-FP8 on MI300X with DP8+EP8
AMD · 14 oct. 2026
Principled Top-$k$ Selection for Language Models with Hybrid Gradients
arXiv · 6 oct. 2026