aiqbench analysis
This research proposes spectral weight decay, a novel regularization technique that leverages the nuclear norm of weight matrices to induce low-rank structures in neural networks. By decoupling this update from standard gradient steps, the method significantly enhances model compression and inference speed on LLaMA architectures without sacrificing validation loss. The approach also demonstrates robustness against high levels of label noise, offering a more efficient alternative to conventional $\ell_2$ regularization for large-scale language models.
Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
arXiv:2610.11730v1 Announce Type: new Abstract: Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.
Abstract:Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at this https URL.
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2610.11730 [cs.LG]
(or arXiv:2610.11730v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.11730 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 244 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Singular Bayesian Neural Networks
arXiv · Jul 21, 2026
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
arXiv · Sep 24, 2026
Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic
arXiv · Jun 9, 2026