analiza aiqbench
DADP introduces a biologically inspired pruning technique that dynamically allocates sparsity during training based on synaptic activity, eliminating the need for manual per-layer budgeting. By leveraging accumulated activation and gradient products, this method effectively reduces compute overhead while preserving latent feature diversity even at extreme sparsity levels. The approach demonstrates robust performance across various architectures without inducing representation collapse, offering a practical solution to the redundancy inherent in over-parameterized neural networks.
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
arXiv:2610.11853v1 Announce Type: new Abstract: Modern neural networks are heavily over-parameterized. This redundancy incurs substantial compute and memory overhead during training and inference. Existing pruning methods rely on post-hoc magnitude thresholds or static initialization heuristics. Consequently, they often require manual per-layer sparsity targets or expensive retraining cycles. We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism. During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients. Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning. Across MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT architectures, DADP matches or outperforms Magnitude, SNIP and RigL, retaining 73.67% accuracy (dense baseline: 76.06%) at 99% sparsity on ResNet-18. Finally, matrix-based Shannon entropy and effective rank measurements confirm that DADP preserves latent feature diversity at extreme sparsities without representation collapse.
Abstract:Modern neural networks are heavily over-parameterized. This redundancy incurs substantial compute and memory overhead during training and inference. Existing pruning methods rely on post-hoc magnitude thresholds or static initialization heuristics. Consequently, they often require manual per-layer sparsity targets or expensive retraining cycles. We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism. During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients. Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning. Across MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT architectures, DADP matches or outperforms Magnitude, SNIP and RigL, retaining 73.67% accuracy (dense baseline: 76.06%) at 99% sparsity on ResNet-18. Finally, matrix-based Shannon entropy and effective rank measurements confirm that DADP preserves latent feature diversity at extreme sparsities without representation collapse.
Machine Learning (cs.LG)
arXiv:2610.11853 [cs.LG]
(or arXiv:2610.11853v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.11853 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 237 cuvinte
Subiecte conexe
[AINews] Reflection Beam - 501B-A23B American Open Model
Latent Space · 6 oct. 2026
Molybdenum disulfide transistors reshape spiking neural networks
Quantum Zeitgeist · 14 sept. 2026
Does preservation make sense before we know how to revive?
LessWrong · 15 iun. 2026
What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
EleutherAI · 25 aug. 2026