analiza aiqbench
This is an arXiv preprint, so its results are unreviewed and rest on the authors' own fitted scaling law. The core claim is that mixture-of-experts models can absorb the cost of adding languages more efficiently than dense models, with 32 experts cutting one reported loss penalty by 83%. Readers building multilingual systems should look at the held-out evaluation and the matched-compute setup before treating the 28% and 83% figures as general guidance.
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
arXiv:2610.20247v1 Announce Type: new Abstract: Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
Abstract:Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.20247 [cs.CL]
(or arXiv:2610.20247v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.20247 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 244 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
arXiv · 28 iul. 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · 24 iul. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026