aiqbench analysis
This research investigates how transformer layers geometrically reorganize token representations, revealing distinct dimensionality trajectories for closed-class versus open-class words. By analyzing intrinsic dimensionality and neighborhood structures across encoder and decoder architectures, the study demonstrates that grammatical roles can be recovered purely from these geometric features. These findings offer critical insights into the internal mechanics of large language models, specifically how they compress and structure semantic information during context integration.
The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers
arXiv:2608.25166v1 Announce Type: new Abstract: Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.
Abstract:Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.
Computation and Language (cs.CL)
arXiv:2608.25166 [cs.CL]
(or arXiv:2608.25166v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.25166 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 276 words
Related coverage
Decomposing how prompting steers behavior
arXiv · Jun 3, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Running Variational Quantum Eigensolver with Qiskit Aer on AMD Instinct
AMD · May 29, 2026
How Does Changing the Elo of a Chess Transformer Affect its Computations?
LessWrong · Sep 29, 2026