aiqbench analysis
This study extends theoretical scaling law analysis from linear models to quadratic and diagonal networks, revealing how spectral properties of weights govern generalization in the feature learning regime. By linking power-law tails in the weight spectrum to excess risk through connections with compressed sensing, the work provides a first-principles explanation for empirical phenomena observed in deep learning. These findings offer a rigorous mathematical foundation for understanding the crossover behaviors and plateaus that characterize modern neural network performance.
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv:2509.24882v2 Announce Type: replace-cross Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.
Abstract:Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.
Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
arXiv:2509.24882 [cs.LG]
(or arXiv:2509.24882v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2509.24882 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 255 words
Related coverage
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv · Aug 7, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
arXiv · Jul 20, 2026
A Theory of Saddle Escape in Deep Nonlinear Networks
arXiv · Sep 30, 2026