aiqbench analysis
The paper finds that ordered vocabulary logits in language models follow a log-linear decay across contexts, architectures, and scales, with the slope rising during pretraining. Simple order-statistic baselines explain the shape but not the higher trained slope (about 1.4 to 1.7 versus roughly 0.2 at initialization), and the authors report that slope correlates with predictive performance. For practitioners, this is a diagnostic, not yet a validated mechanism, and its value depends on whether the slope predicts quality beyond existing metrics and whether it can inform decoding or calibration.
The Geometry of Language Model Logits: Generic Structure and Learned Departures
arXiv:2610.18520v1 Announce Type: cross Abstract: Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation \[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.
Abstract:Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation \[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.18520 [cs.CL]
(or arXiv:2610.18520v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.18520 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 234 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
When Partial Credit Opposes Completion
Snorkel AI Blog · Oct 15, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?
arXiv · Sep 30, 2026