analiza aiqbench
Attention pruning is a practical efficiency lever: the authors add an input-dependent bias to the query-key product that scores each token's expected influence, and use that score to drop attention-map entries. Their reported result is lower FLOPs and latency with most classification accuracy kept, but the evidence is limited to vision transformers, and the accuracy trade-off and speedups still need checking against the released code and benchmarks. Readers working on inference cost should watch whether the token-importance proxy holds up on language models and larger architectures.
Exploiting Biases in Attention for Transformers
arXiv:2611.04657v1 Announce Type: new Abstract: Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at https://github.com/EIDOSLAB/AttentionWithBias.
Abstract:Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at this https URL.
Machine Learning (cs.LG)
arXiv:2611.04657 [cs.LG]
(or arXiv:2611.04657v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.04657 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
arXiv · 26 sept. 2026
Vision And Text Transformer For Predicting Answerability On Visual Question Answering
arXiv · 16 sept. 2026
TIDFormer: Exploiting Temporal and Interactive Dynamics Makes A Great Dynamic Graph Transformer
arXiv · 3 iun. 2026
Noise-Directed Warm-Starting Boosts 100-Qubit Quantum Optimization
Quantum Zeitgeist · 13 iul. 2026