analiza aiqbench
This research reveals a fundamental theoretical divide between unstructured weight pruning and structured neuron pruning within the context of the Strong Lottery Ticket Hypothesis. The study demonstrates that approximating target functions requires exponentially larger initial networks for neuron-based methods compared to weight-level sparsification, challenging assumptions about hardware-efficient model compression without training.
Structured vs. Unstructured Pruning: An Exponential Gap
arXiv:2603.02234v3 Announce Type: replace-cross Abstract: The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient. Pruning methods are typically classified as unstructured, where individual weights can be removed from the network, and structured, where parameters are removed according to specific patterns, as in neuron pruning. Existing theoretical results supporting the SLTH rely almost exclusively on unstructured pruning, showing that logarithmic overparameterization suffices to approximate simple target networks. In contrast, neuron pruning has received limited theoretical attention, despite its practical appeal for direct hardware speedups. In this work, we consider the problem of approximating a single bias-free ReLU neuron by pruning hidden units of a randomly initialized two-layer ReLU network, effectively isolating the intrinsic limitations of neuron pruning. We show that achieving an $\varepsilon$-approximation requires a starting network size of $\Omega(1/\varepsilon)$ for neuron pruning, whereas weight pruning succeeds with only $O(\log(1/\varepsilon))$ hidden units, revealing an exponential separation between the two approaches.
Abstract:The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient. Pruning methods are typically classified as unstructured, where individual weights can be removed from the network, and structured, where parameters are removed according to specific patterns, as in neuron pruning. Existing theoretical results supporting the SLTH rely almost exclusively on unstructured pruning, showing that logarithmic overparameterization suffices to approximate simple target networks. In contrast, neuron pruning has received limited theoretical attention, despite its practical appeal for direct hardware speedups. In this work, we consider the problem of approximating a single bias-free ReLU neuron by pruning hidden units of a randomly initialized two-layer ReLU network, effectively isolating the intrinsic limitations of neuron pruning. We show that achieving an $\varepsilon$-approximation requires a starting network size of $\Omega(1/\varepsilon)$ for neuron pruning, whereas weight pruning succeeds with only $O(\log(1/\varepsilon))$ hidden units, revealing an exponential separation between the two approaches.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2603.02234 [cs.LG]
(or arXiv:2603.02234v3 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 181 cuvinte
Subiecte conexe
Reflections and Fragments: Securing LLMs Against Sequential Mosaic Attacks
arXiv · 7 oct. 2026
Balancing Expressivity and Overfitting in Quantum Gaussian Process Regression
arXiv · 10 sept. 2026
Researchers Limit Overfitting with Tuned Quantum Machine Learning
Quantum Zeitgeist · 1 oct. 2026
Quantum Speedups Require Structure or Depth
arXiv · 20 aug. 2026