What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Title:What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Abstract:Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position zero, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the P0-Sink Circuit, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at this https URL.

Machine Learning (cs.LG); Computation and Language (cs.CL)

arXiv:2603.06591 [cs.LG]

(or arXiv:2603.06591v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.06591 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.