aiqbench analysis
This paper is a machine learning research study on multilingual pretraining, so it passes the topic gate. Its primary subject is how to structure training sequences for large language models, specifically whether concatenating parallel translations helps cross-lingual performance. The main takeaway for practitioners is that parallel-context pretraining alone doesn't reliably beat standard pretraining, but a two-stage scheme (parallel first, then monolingual) improves results at matched token budgets, though the evidence is limited to the reported settings and benchmarks.
How to Use Parallel Data for Multilingual Pretraining
arXiv:2610.28838v1 Announce Type: cross Abstract: High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its translation in the same training sequence. We compare this parallel pretraining (PPT) approach with standard pretraining (SPT), training models from scratch at matched token budgets. Surprisingly, despite the benefits of parallel data reported in prior cross-lingual work, PPT alone does not consistently improve language understanding over SPT. Our further analysis suggests that although PPT builds a strong cross-lingual foundation, as shown by its better use of shared information across languages, the mismatch between parallel training contexts and single-language task inputs limits its gains in general language understanding. This finding motivates a two-stage approach following the pretraining-fine-tuning paradigm: we first use PPT to build a cross-lingual foundation, then continue with SPT to adapt it to standard language modeling. In both high-resource and low-resource settings, the two-stage approach consistently improves general language understanding and cross-lingual task performance over standard pretraining alone at matched training budgets.
Abstract:High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its translation in the same training sequence. We compare this parallel pretraining (PPT) approach with standard pretraining (SPT), training models from scratch at matched token budgets. Surprisingly, despite the benefits of parallel data reported in prior cross-lingual work, PPT alone does not consistently improve language understanding over SPT. Our further analysis suggests that although PPT builds a strong cross-lingual foundation, as shown by its better use of shared information across languages, the mismatch between parallel training contexts and single-language task inputs limits its gains in general language understanding. This finding motivates a two-stage approach following the pretraining-fine-tuning paradigm: we first use PPT to build a cross-lingual foundation, then continue with SPT to adapt it to standard language modeling. In both high-resource and low-resource settings, the two-stage approach consistently improves general language understanding and cross-lingual task performance over standard pretraining alone at matched training budgets.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.28838 [cs.CL]
(or arXiv:2610.28838v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.28838 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 309 words
Related coverage
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
arXiv · Sep 22, 2026
Scaling Multilingual Pretraining of Pixel Language Models
arXiv · Oct 22, 2026
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
arXiv · Oct 12, 2026
One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
arXiv · Aug 27, 2026