aiqbench analysis
This paper is squarely about machine learning: a neural network pruning method (Subspace Node Pruning) that removes neurons, filters, attention heads, or layers and recovers lost accuracy via linear least squares, with no retraining afterward. Its evidence covers CNNs on ImageNet and LLMs such as OPT-125M and LLaMA-2 7B, with gains concentrated in low-shot regimes. Readers should treat the results as benchmark-specific comparisons against LLM Surgeon and other baselines, but the core idea of unifying importance measures under a quadratic form is a useful reference for efficient inference work.
Subspace Node Pruning
arXiv:2405.17506v4 Announce Type: replace Abstract: Node pruning is the art of removing computational units such as neurons, filters, attention heads, or even entire layers to significantly reduce inference time while retaining network performance. In this work, we show that a broad family of importance measures, from activation-based to Fisher-based, share a common quadratic form. Through a lower-triangular change of basis, this form yields a pruning subspace in which entire nodes can be removed while the impact of the lost units is simultaneously recovered via linear least squares. Furthermore, the importance measures within our subspace are decoupled and additive through an orthogonalization of the layer activations, allowing multi-node pruning within layers without interference between importance scores. Our method, Subspace Node Pruning (SNP), outperforms the compared baselines on VGG-16 and ResNet-50 on ImageNet. On OPT-125M and LLaMA-2 7B, SNP matches the state-of-the-art LLM Surgeon at a 40-shot schedule and outperforms it whenever fewer shots are used, all without any retraining after pruning.
Abstract:Node pruning is the art of removing computational units such as neurons, filters, attention heads, or even entire layers to significantly reduce inference time while retaining network performance. In this work, we show that a broad family of importance measures, from activation-based to Fisher-based, share a common quadratic form. Through a lower-triangular change of basis, this form yields a pruning subspace in which entire nodes can be removed while the impact of the lost units is simultaneously recovered via linear least squares. Furthermore, the importance measures within our subspace are decoupled and additive through an orthogonalization of the layer activations, allowing multi-node pruning within layers without interference between importance scores. Our method, Subspace Node Pruning (SNP), outperforms the compared baselines on VGG-16 and ResNet-50 on ImageNet. On OPT-125M and LLaMA-2 7B, SNP matches the state-of-the-art LLM Surgeon at a 40-shot schedule and outperforms it whenever fewer shots are used, all without any retraining after pruning.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
arXiv:2405.17506 [cs.LG]
(or arXiv:2405.17506v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2405.17506 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 258 words
Related coverage
Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models
arXiv · Nov 2, 2026
Hybrid-Field Sparse Channel Representation and Recovery for XL-RIS-Assisted mmWave MIMO Systems
arXiv · Aug 4, 2026
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
arXiv · Oct 9, 2026
Nullspace-guided Adaptive Bootstrap of Quantum Many-body Systems
arXiv · Sep 23, 2026