analiza aiqbench
SSA addresses the performance degradation and capability gaps inherent in sparse attention by aligning full and sparse outputs in feature space, effectively mitigating train-inference distribution mismatches. The method theoretically bounds approximation error based on dropped attention mass and empirically achieves state-of-the-art results with superior long-context handling across varying sparsity budgets. This framework offers a practical path to reducing quadratic complexity without sacrificing the representational power of full self-attention.
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
arXiv:2511.20102v4 Announce Type: replace Abstract: Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.
Abstract:Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.
Computation and Language (cs.CL)
arXiv:2511.20102 [cs.CL]
(or arXiv:2511.20102v4 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 137 cuvinte
Subiecte conexe
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
arXiv · 4 iun. 2026
Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers
arXiv · 4 iun. 2026
Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers
arXiv · 8 iun. 2026
Self-Explaining Hate Speech Detection with Moral Rationales
arXiv · 21 iul. 2026