analiza aiqbench
This research investigates the efficiency trade-offs of Mixture-of-Experts architectures specifically within particle-physics transformers, finding that top-1 routing improves accuracy over dense baselines without increasing active compute. The study highlights that simply adding stored experts yields diminishing returns, while activating multiple experts per token requires a higher computational cost for marginal gains. These findings provide critical insights into optimizing sparse expert models for high-energy physics classification tasks where parameter capacity and active computation must be carefully distinguished.
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
arXiv:2610.02701v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models can increase parameter capacity without proportionally increasing active computation, but it is unclear how this trade-off behaves in particle-physics transformers. We study dense and MoE Particle Transformers on 188-class JetClass-II, varying expert count, routing capacity, top-K, and auxiliary loss. We find that, when token dropping is avoided, top-1 MoE models improve over the dense baseline at nearly unchanged nominal forward compute, while further increasing the number of stored experts produces little additional accuracy gain. Activating multiple experts per token yields additional predictive improvements at higher computational cost. Routing analyses show that expert assignments become more strongly associated with particle identity and kinematics in some configurations, but this structure does not increase monotonically with classification performance. These results highlight the need to distinguish stored parameter capacity, active computation, routing capacity, and routing organization when evaluating sparse expert models for jet classification. Code and experiment configurations are available at https://github.com/kpendiyala/MPT.
Authors:Kaushik Pendiyala, Haris Zia, Trevin Lee, Timothy Legge, Alejandro J. De Leon, Zihan Zhao, Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Richard Cavanaugh, Javier Duarte
Abstract:Mixture-of-Experts (MoE) models can increase parameter capacity without proportionally increasing active computation, but it is unclear how this trade-off behaves in particle-physics transformers. We study dense and MoE Particle Transformers on 188-class JetClass-II, varying expert count, routing capacity, top-K, and auxiliary loss. We find that, when token dropping is avoided, top-1 MoE models improve over the dense baseline at nearly unchanged nominal forward compute, while further increasing the number of stored experts produces little additional accuracy gain. Activating multiple experts per token yields additional predictive improvements at higher computational cost. Routing analyses show that expert assignments become more strongly associated with particle identity and kinematics in some configurations, but this structure does not increase monotonically with classification performance. These results highlight the need to distinguish stored parameter capacity, active computation, routing capacity, and routing organization when evaluating sparse expert models for jet classification. Code and experiment configurations are available at this https URL.
8 pages, 4 figures. Submitted to the ML4PS 2026 workshop
Machine Learning (cs.LG); High Energy Physics - Experiment (hep-ex)
arXiv:2610.02701 [cs.LG]
(or arXiv:2610.02701v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.02701 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 211 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
NVIDIA · 14 sept. 2026
NVIDIA boosts MoE AI training 10.4x with JAX and Transformer Engine
Quantum Zeitgeist · 15 sept. 2026
MoRE: Mixture of Reused Experts
arXiv · 17 sept. 2026