analiza aiqbench
This research introduces Weight-Redundancy Pruning (WRP), a novel technique that eliminates the need for calibration data or forward passes by leveraging checkpoint weights to identify redundant Transformer blocks. By estimating inter-layer redundancy through attention outputs and MLP projections, WRP achieves inference cost reductions comparable to activation-based methods while outperforming existing magnitude pruning approaches. The method demonstrates robust performance across diverse model families and tasks, offering a practical pathway for optimizing LLM efficiency without additional data overhead.
Forward-Free LLM Depth Pruning via Weight Redundancy
arXiv:2609.09883v1 Announce Type: cross Abstract: Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.
Abstract:Depth pruning reduces large language model (LLM) inference cost by removing complete Transformer blocks. Activation-based methods collect hidden states through forward passes on calibration data, while existing forward-free methods score each Transformer block separately without measuring similarity between blocks. We propose Weight-Redundancy Pruning (WRP), a forward-free depth-pruning method that estimates inter-layer redundancy from checkpoint weights to select blocks without calibration data or model forward passes. WRP compares attention output and MLP down-projection weights across layers and combines their pairwise similarities with relative projection-scale information. The resulting all-pairs similarity matrix guides layer grouping and block selection. Across multiple pruning settings, model families, and downstream tasks, WRP consistently outperforms existing forward-free magnitude pruning and approaches the performance of activation-based methods.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF)
arXiv:2609.09883 [cs.LG]
(or arXiv:2609.09883v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.09883 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 217 cuvinte
Subiecte conexe
Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus
AMD · 3 sept. 2026
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
arXiv · 9 iun. 2026
From Patching to Pruning Visual Computation in Vision Language Models
arXiv · 5 oct. 2026
Quantization and Pruning Methods to Make Your LLM Leaner
KDnuggets · 28 aug. 2026