aiqbench analysis
This arXiv paper studies weight perturbation, a gradient-free learning method that updates parameters using random perturbations and scalar loss changes instead of backpropagation, and benchmarks it on fully connected MNIST networks. The reported numbers show near-BP accuracy only under large batches and long training, with a clear efficiency gap (90.2% vs. 97.4% at ten epochs), and the theoretical approximation argument is conditional on stated regularity assumptions rather than a general guarantee. For ML readers, the practical takeaway is that gradient-free training remains a niche research direction whose viability depends heavily on compute budget and network depth, not a drop-in replacement for backprop.
Approaching Backpropagation Accuracy with Weight Perturbation on MNIST
arXiv:2610.21665v1 Announce Type: new Abstract: Weight perturbation (WP) estimates learning directions from random parameter perturbations and a scalar change in loss, without backpropagating derivatives. We examine how batch size, training duration, and estimator stabilization affect its attainable accuracy in fully connected MNIST networks. In the shallow configuration, large-batch training reaches approximately 97\% test accuracy, compared with an early backpropagation (BP) reference near 97.5--98\%. The same comparison exposes a substantial efficiency gap: under a ten-epoch budget, shallow WP reaches 90.2\%, whereas the reported BP reference reaches 97.4\%. Epoch-indexed curves show diminishing returns from increasing batch size beyond 4,000. Learning-rate reduction and averaging independent perturbation-based updates also stabilize deeper networks, although their recorded results do not establish BP-level accuracy. We distinguish these empirical findings from a finite-horizon approximation argument: under stated regularity conditions, population WP can approximate a fixed gradient-descent trajectory arbitrarily closely at any fixed finite depth. Positive classification margins then suffice to preserve the reference predictions on a finite test set with arbitrarily high probability. Together, the results identify near-BP accuracy as a configuration- and budget-dependent possibility, rather than evidence of equal efficiency or an unconditional accuracy guarantee across depth.
Abstract:Weight perturbation (WP) estimates learning directions from random parameter perturbations and a scalar change in loss, without backpropagating derivatives. We examine how batch size, training duration, and estimator stabilization affect its attainable accuracy in fully connected MNIST networks. In the shallow configuration, large-batch training reaches approximately 97\% test accuracy, compared with an early backpropagation (BP) reference near 97.5--98\%. The same comparison exposes a substantial efficiency gap: under a ten-epoch budget, shallow WP reaches 90.2\%, whereas the reported BP reference reaches 97.4\%. Epoch-indexed curves show diminishing returns from increasing batch size beyond 4,000. Learning-rate reduction and averaging independent perturbation-based updates also stabilize deeper networks, although their recorded results do not establish BP-level accuracy. We distinguish these empirical findings from a finite-horizon approximation argument: under stated regularity conditions, population WP can approximate a fixed gradient-descent trajectory arbitrarily closely at any fixed finite depth. Positive classification margins then suffice to preserve the reference predictions on a finite test set with arbitrarily high probability. Together, the results identify near-BP accuracy as a configuration- and budget-dependent possibility, rather than evidence of equal efficiency or an unconditional accuracy guarantee across depth.
Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
arXiv:2610.21665 [cs.LG]
(or arXiv:2610.21665v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.21665 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 284 words
Related coverage
Sakana AI’s Error Diffusion Trains Dale-Compliant Dual-Stream Networks, Reaching 96.7% MNIST and 61.7% CIFAR-10 Without Backpropagation
MarkTechPost · Jul 18, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Causal Unlearning in Collaborative Optimization: Exact and Approximate Influence Reversal under Adversarial Contributions
arXiv · Jun 9, 2026
Gradient-Free Training of Spiking Neural Networks via Low-Rank Evolution Strategies
arXiv · Sep 9, 2026