aiqbench analysis
This research addresses the multilingual reasoning gap by demonstrating that language-agnostic reasoning capabilities reside in a model's middle layers, distinct from language-specific outer layers. By swapping these mid-layers between English and native-language specialists, the study successfully narrows performance disparities without forcing models to abandon their target language for internal thought processes. The findings offer a practical architectural solution for building more equitable multilingual AI systems that maintain high reasoning standards across diverse languages.
Rethinking the Multilingual Reasoning Gap with Layer Swap
arXiv:2605.26735v2 Announce Type: replace Abstract: Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language (native reasoning) substantially degrades performance relative to allowing the model to reason in English before answering in the input language (English-pivoted reasoning). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of Qwen/Qwen3-8B-Base, and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist's stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets.
Abstract:Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language (native reasoning) substantially degrades performance relative to allowing the model to reason in English before answering in the input language (English-pivoted reasoning). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of Qwen/Qwen3-8B-Base, and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist's stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets.
Computation and Language (cs.CL)
arXiv:2605.26735 [cs.CL]
(or arXiv:2605.26735v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.26735 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 309 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders
arXiv · Aug 27, 2026
Small Language Models with Hugging Face transformers Library + smolLM3
KDnuggets · Aug 7, 2026