analiza aiqbench
This research addresses the "curse of multilinguality" by demonstrating that upscaling model capacity is a more efficient strategy than standard continued pretraining for adapting to new languages. The study reveals a nuanced trade-off where initial English-based upscaling aids low-resource languages, while direct scaling on target data proves optimal for higher-resource scenarios to minimize compute costs and prevent forgetting.
Outgrowing the Curse of Multilinguality: Scaling Strategies for Efficient Language Adaptation
arXiv:2512.10772v2 Announce Type: replace-cross Abstract: Training high-performing language models which include medium- and lower-resource languages remains a challenge. Massively multilingual models still underperform language-specific adaptations, especially at smaller scales, due to the curse of multilinguality. In this work, we investigate scaling as an efficient strategy for adapting pretrained models to new target languages. Through comprehensive ablations matched for data and training FLOPs, we test whether upscaling from English base models enables more effective and resource-efficient adaptation than standard continued pretraining. We find that upscaling is an efficient strategy for improving target-language performance by relieving capacity constraints, especially for higher-resourced languages. An initial upscaling step in English is beneficial for low-resource languages. However, in higher-resource scenarios it is more compute-efficient to directly scale up on the target language, albeit slightly compromising data efficiency for most languages. For multilingual continued pre-training, we identify directly scaling up using the target data as the optimal setup for both data efficiency and reduced English forgetting.
Abstract:Training high-performing language models which include medium- and lower-resource languages remains a challenge. Massively multilingual models still underperform language-specific adaptations, especially at smaller scales, due to the curse of multilinguality. In this work, we investigate scaling as an efficient strategy for adapting pretrained models to new target languages. Through comprehensive ablations matched for data and training FLOPs, we test whether upscaling from English base models enables more effective and resource-efficient adaptation than standard continued pretraining. We find that upscaling is an efficient strategy for improving target-language performance by relieving capacity constraints, especially for higher-resourced languages. An initial upscaling step in English is beneficial for low-resource languages. However, in higher-resource scenarios it is more compute-efficient to directly scale up on the target language, albeit slightly compromising data efficiency for most languages. For multilingual continued pre-training, we identify directly scaling up using the target data as the optimal setup for both data efficiency and reduced English forgetting.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2512.10772 [cs.CL]
(or arXiv:2512.10772v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2512.10772 Focus to learn more arXiv-issued DOI via DataCite
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026, pp. 4329-4344
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 265 cuvinte