analiza aiqbench
Stem addresses the quadratic scaling bottleneck in LLMs by introducing position-dependent sparsity strategies that respect causal dependencies, offering improved efficiency during long-context pre-filling without sacrificing accuracy. This research advances the theoretical understanding of information flow within attention mechanisms and provides a practical module for optimizing model inference on extended sequences.
Stem: Rethinking Causal Information Flow in Sparse Attention
arXiv:2603.06274v2 Announce Type: replace Abstract: The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-filling phase. In this paper, we rethink the causal attention mechanism from the perspective of information flow. Due to causal constraints, tokens at initial positions participate in the aggregation of every subsequent token. However, existing sparse methods typically apply a uniform top-k selection across all token positions within a layer, ignoring the cumulative dependency of token information inherent in causal architectures. To address this, we propose Stem, a novel, plug-and-play sparsity module aligned with information flow. First, Stem employs the Token Position-Decay strategy, applying position-dependent top-k within each layer to retain initial tokens for recursive dependencies. Second, to preserve information-rich tokens, Stem utilizes the Output-Aware Metric. It prioritizes high-impact tokens based on approximate output magnitude. Extensive evaluations demonstrate that Stem achieves superior accuracy with reduced computation and pre-filling latency.
Abstract:The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-filling phase. In this paper, we rethink the causal attention mechanism from the perspective of information flow. Due to causal constraints, tokens at initial positions participate in the aggregation of every subsequent token. However, existing sparse methods typically apply a uniform top-k selection across all token positions within a layer, ignoring the cumulative dependency of token information inherent in causal architectures. To address this, we propose Stem, a novel, plug-and-play sparsity module aligned with information flow. First, Stem employs the Token Position-Decay strategy, applying position-dependent top-k within each layer to retain initial tokens for recursive dependencies. Second, to preserve information-rich tokens, Stem utilizes the Output-Aware Metric. It prioritizes high-impact tokens based on approximate output magnitude. Extensive evaluations demonstrate that Stem achieves superior accuracy with reduced computation and pre-filling latency.
Accepted at ICML 2026. Lin Niu and Xin Luo contributed equally to this work. Camera-ready version
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.06274 [cs.LG]
(or arXiv:2603.06274v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.06274 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 262 cuvinte
Subiecte conexe
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Triton-Based Optimization of Video Sparse Attention on ROCm
AMD · 13 iul. 2026
HodgePhaseNet: Discrete Hodge Decomposition of Semantic Flows in Transformer Attention
arXiv · 14 oct. 2026
Transformers Learn the Mestre-Nagao Heuristic
arXiv · 16 iun. 2026