aiqbench analysis
This research investigates how lexical diversity in training data creates distinct regimes of intrinsic dimensionality within language model representations, challenging the assumption that ID directly correlates with linguistic complexity alone. By deriving a parameter-free formula for the transition between these regimes, the study reveals fundamental structural principles governing how LLMs organize high-dimensional manifolds based on dataset construction artifacts rather than inherent properties of language itself.
Local and Global Regimes of Geometric Complexity in Language Model Representations
arXiv:2608.14361v1 Announce Type: new Abstract: Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.
Abstract:Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.
Computation and Language (cs.CL)
arXiv:2608.14361 [cs.CL]
(or arXiv:2608.14361v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.14361 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 278 words
Related coverage
Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability
LessWrong · Jul 25, 2026
A Deep Dive into LDS Optimizations on AMD Instinct MI450 GPUs
AMD · Aug 28, 2026
Integrating Local and Global Entropy for Uncertainty Quantification in LLMs
arXiv · Jun 10, 2026
Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training
arXiv · Sep 1, 2026