aiqbench analysis
This paper offers a theoretical explanation for the "Platonic representation hypothesis" in multilingual language models by proposing that data possess a hidden hierarchical structure shared across languages at abstract levels. The authors demonstrate that belief propagation on synthetic grammars yields analytical predictions matching transformer behavior, clarifying why cross-lingual similarity peaks in middle layers and distinguishing it from coordinate alignment. These findings provide critical insight into the internal geometry of LLMs and how they generalize linguistic concepts independent of surface modality.
A theory of platonic representations in language models
arXiv:2610.07168v1 Announce Type: cross Abstract: Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
Abstract:Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
Machine Learning (cs.LG); Computation and Language (cs.CL); Machine Learning (stat.ML)
arXiv:2610.07168 [cs.LG]
(or arXiv:2610.07168v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.07168 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 275 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
What can language models teach us about understanding?
LessWrong · Oct 9, 2026
From Plato to Aristotle: Toward a General Theory of Representation Hypotheses
arXiv · Oct 15, 2026
Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities
arXiv · Sep 22, 2026