aiqbench analysis
This research challenges the prevailing assumption that multilingual LLMs isolate language-specific information solely in their initial and final layers, demonstrating instead that linguistic identity is distributed throughout the model's architecture. Using fine-grained analysis of Portuguese and Spanish representations alongside a large-scale experiment across over 200 languages, the study reveals that intermediate layers actively participate in encoding language features rather than supporting purely language-agnostic processing. Additionally, the findings highlight a systematic skew in model behavior where ambiguous inputs are biased toward Spanish classifications, diverging from human judgment patterns.
Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
arXiv:2610.25505v1 Announce Type: new Abstract: Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Title:Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
Abstract:Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Computation and Language (cs.CL)
arXiv:2610.25505 [cs.CL]
(or arXiv:2610.25505v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.25505 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 245 words