analiza aiqbench
BudgetFormer introduces a dynamic mechanism that allocates attention heads based on input complexity, significantly reducing FLOPs and memory usage for tasks like text classification without sacrificing accuracy. By learning a head budget and relevance distribution, the architecture effectively prunes unnecessary computation while maintaining performance parity with standard multi-head attention models. This approach offers a practical pathway for optimizing Transformer efficiency in scenarios where global information suffices for coarse-grained tasks.
Adaptive Head Budgeting for Efficient Multi-Head Attention
arXiv:2604.22583v3 Announce Type: replace Abstract: Multi-head attention enables Transformers to capture diverse representations, but all attention heads are typically activated for every input, regardless of task complexity. For coarse-grained tasks such as text classification, where relevant information is often global, this fixed allocation can introduce unnecessary computation. We propose BudgetFormer, a Transformer architecture that dynamically allocates attention heads on a per-input basis. The model learns both a head budget and a relevance distribution to select the most informative heads. To support effective head selection, we introduce a training strategy that balances exploration and exploitation. Experiments on text classification tasks show that BudgetFormer reduces FLOPs and memory usage while matching or surpassing the performance of standard multi-head attention. These results highlight adaptive head allocation as an effective approach to improving Transformer efficiency and performance.
Abstract:Multi-head attention enables Transformers to capture diverse representations, but all attention heads are typically activated for every input, regardless of task complexity. For coarse-grained tasks such as text classification, where relevant information is often global, this fixed allocation can introduce unnecessary computation. We propose BudgetFormer, a Transformer architecture that dynamically allocates attention heads on a per-input basis. The model learns both a head budget and a relevance distribution to select the most informative heads. To support effective head selection, we introduce a training strategy that balances exploration and exploitation. Experiments on text classification tasks show that BudgetFormer reduces FLOPs and memory usage while matching or surpassing the performance of standard multi-head attention. These results highlight adaptive head allocation as an effective approach to improving Transformer efficiency and performance.
Machine Learning (cs.LG)
arXiv:2604.22583 [cs.LG]
(or arXiv:2604.22583v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 139 cuvinte
Subiecte conexe
Adaptive Head Budgeting for Efficient Multi-Head Attention
arXiv · 4 iun. 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · 21 iul. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026