analiza aiqbench
This research introduces Gate-Norm, a novel method for data-free pruning of self-attention layers in large language models that eliminates the need for calibration datasets or fine-tuning. By leveraging the Attention Suppression Hypothesis to identify and remove redundant sublayers based on query-key coupling, the technique achieves significant inference speedups while maintaining accuracy comparable to traditional data-driven approaches. The primary contribution lies in demonstrating a highly efficient pathway for compressing LLMs without sacrificing performance, addressing critical deployment constraints.
Data-Free Pruning of Self-Attention Layers in LLMs
arXiv:2512.20636v2 Announce Type: replace Abstract: Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query-key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning 8-16 attention sublayers yields up to $1.30\times$ higher inference throughput while keeping average zero-shot accuracy within 1.5 percentage points of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being $\sim 1000\times$ faster to score layers, enabling practical, data-free compression of LLMs.
Abstract:Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query-key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning 8-16 attention sublayers yields up to $1.30\times$ higher inference throughput while keeping average zero-shot accuracy within 1.5 percentage points of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being $\sim 1000\times$ faster to score layers, enabling practical, data-free compression of LLMs.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2512.20636 [cs.LG]
(or arXiv:2512.20636v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2512.20636 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 244 cuvinte
Subiecte conexe
Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
arXiv · 14 oct. 2026
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta Engineering · 3 aug. 2026
Introducing North Mini Code: Cohere’s First Model For Developers
Hugging Face · 9 iun. 2026
PagedAttention vs. RadixAttention: Optimizing LLM KV Cache Management
Analytics Vidhya · 21 aug. 2026