aiqbench analysis
This research introduces Selective Attention Freezing (SAF), a technique that statically defines specific attention heads by freezing their weights halfway through training, thereby reducing computational costs and storage requirements. By representing these fixed patterns with linear complexity rather than quadratic, the method accelerates optimizer updates and long-context processing with only marginal increases in perplexity. The approach demonstrates that reusing learned attention patterns can improve efficiency without significantly degrading model performance on standard tasks.
When Can Attention Heads Be Statically Defined?
arXiv:2609.34650v1 Announce Type: new Abstract: Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through controlled pretraining comparisons, we identify Selective Attention Freezing (SAF), which selects heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means halfway through training. We represent these fixed patterns with absolute-position and relative-distance preferences, reducing storage from quadratic to linear in sequence length. A fused kernel reconstructs the patterns and executes ordinary-attention and replaced heads together. At matched training-token budgets, replacing 25% of attention heads gives 1.056x faster post-replacement optimiser updates at 124M parameters and 4K context, with a 0.77% perplexity increase. At 1B and 8K context, post-replacement updates are 1.068x faster on four GPUs including communication, with a 0.51% perplexity increase. The resulting models also accelerate long-input finetuning and causal prefill. After associative-recall adaptation, the 124M model with 25% replacement generalises to more key-value pairs at a fixed length better than ordinary attention and two pruning controls.
Title:When Can Attention Heads Be Statically Defined?
Abstract:Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through controlled pretraining comparisons, we identify Selective Attention Freezing (SAF), which selects heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means halfway through training. We represent these fixed patterns with absolute-position and relative-distance preferences, reducing storage from quadratic to linear in sequence length. A fused kernel reconstructs the patterns and executes ordinary-attention and replaced heads together. At matched training-token budgets, replacing 25% of attention heads gives 1.056x faster post-replacement optimiser updates at 124M parameters and 4K context, with a 0.77% perplexity increase. At 1B and 8K context, post-replacement updates are 1.068x faster on four GPUs including communication, with a 0.51% perplexity increase. The resulting models also accelerate long-input finetuning and causal prefill. After associative-recall adaptation, the 124M model with 25% replacement generalises to more key-value pairs at a fixed length better than ordinary attention and two pruning controls.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.34650 [cs.LG]
(or arXiv:2609.34650v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.34650 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 271 words
Related coverage
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
arXiv · Sep 30, 2026
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
arXiv · Jun 9, 2026
Presentation: Adaptive Recommenders in the Real World: Inference, Evals, and System Design
InfoQ · Sep 26, 2026