analiza aiqbench
This arXiv paper is a theoretical study of attention sinks and diagonal patterns in transformers, examining when they enable hard attention "switching" and how they prevent oversmoothing (token representations collapsing together with depth). Its main contributions are geometric conditions for representing sinks, an equivalence between sinks and hard attention switches, and a cost comparison explaining why pretrained models favor sinks over diagonal patterns. For ML researchers, this is a useful mechanistic account of why transformers behave as they do, though the empirical evidence is limited to verifying conditions "in practice" rather than demonstrating downstream performance gains.
Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
arXiv:2605.08453v2 Announce Type: replace Abstract: This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention is expected to smooth more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs.\ diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.
Title:Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention
Abstract:This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention is expected to smooth more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs.\ diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2605.08453 [cs.LG]
(or arXiv:2605.08453v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.08453 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 288 cuvinte
Subiecte conexe
Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks
arXiv · 31 iul. 2026
Fractional Heat Kernel for Semi-Supervised Graph Learning with Small Training Sample Size
arXiv · 9 oct. 2026
Geometry-Aware Operator Families for Structured Representation Learning
arXiv · 30 sept. 2026
SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting
arXiv · 18 sept. 2026