aiqbench analysis
This study identifies a measurable spectral reorganization in hidden states that precedes the formation of attention sinks, offering an early diagnostic signal beyond standard activation metrics. By causally perturbing this specific geometry, researchers can modulate subsequent sink behavior across multiple large language models without relying on network depth as a proxy. The findings advance the theoretical understanding of internal model dynamics and provide actionable levers for mitigating pathological attention patterns during inference or training.
Before Attention Sinks Form: Hidden-State Geometry Reveals and Modulates Their Formation
arXiv:2610.16745v1 Announce Type: cross Abstract: Attention sinks are usually studied after they have formed. We ask whether a measurable hidden-state change appears beforehand and whether changing that geometry alters what happens later. To remove network depth as a shortcut, we compare prompts in the same model and at the same layer. When sink timing varies across prompts, simple spectral summaries of the hidden states add information beyond current attention, with the clearest prompt-specific signal in a short window about two to three layers before sink formation. The predictive strength varies across cohorts, so we treat the spectrum as a readout of a pre-sink state change rather than a universal warning statistic. The signal is broader than one measured extreme activation. We then modify only the singular values of the post-layer hidden states after current attention has already been computed. These interventions change later sink behavior in Qwen2.5, GLM-Z1, and GPT-OSS relative to norm-matched random spectral perturbations. In Qwen2.5, the immediate suppressive effect is accompanied by a reduction in next-layer query--key preference for position~0, and the learned spectral directions remain stable when estimated from substantially less training data. Together, the results show that attention sinks are preceded by a short spectral reorganization of hidden states, and directly perturbing this geometry can causally alter subsequent sink formation.
Abstract:Attention sinks are usually studied after they have formed. We ask whether a measurable hidden-state change appears beforehand and whether changing that geometry alters what happens later. To remove network depth as a shortcut, we compare prompts in the same model and at the same layer. When sink timing varies across prompts, simple spectral summaries of the hidden states add information beyond current attention, with the clearest prompt-specific signal in a short window about two to three layers before sink formation. The predictive strength varies across cohorts, so we treat the spectrum as a readout of a pre-sink state change rather than a universal warning statistic. The signal is broader than one measured extreme activation. We then modify only the singular values of the post-layer hidden states after current attention has already been computed. These interventions change later sink behavior in Qwen2.5, GLM-Z1, and GPT-OSS relative to norm-matched random spectral perturbations. In Qwen2.5, the immediate suppressive effect is accompanied by a reduction in next-layer query--key preference for position~0, and the learned spectral directions remain stable when estimated from substantially less training data. Together, the results show that attention sinks are preceded by a short spectral reorganization of hidden states, and directly perturbing this geometry can causally alter subsequent sink formation.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.16745 [cs.LG]
(or arXiv:2610.16745v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.16745 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 307 words
Related coverage
Game Design Creativity evals
LessWrong · Oct 12, 2026
Beyond Dexterity: Why Contact May Define the Next Era of Robotics
IEEE Spectrum · Jun 9, 2026
Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds
arXiv · Aug 19, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026