analiza aiqbench
This arXiv preprint offers a theoretical account of spectral bias, the tendency of neural networks to learn low-frequency components before high-frequency ones, and claims to extend earlier explanations to general activation functions like sigmoid and ReLU with weaker data assumptions. Its core method analyzes the eigenvalue decay and eigenvector shapes of a gradient-descent dynamics matrix, so readers should judge it on whether those derivations hold up and whether the framework yields testable predictions beyond shallow networks. It is a preprint and has not been peer reviewed.
Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
arXiv:2610.21964v1 Announce Type: cross Abstract: Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.
Abstract:Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.
15 pages, 0 figures. The main draft dates back to 2019, with a new abstract and introduction section
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Spectral Theory (math.SP); Machine Learning (stat.ML)
arXiv:2610.21964 [cs.LG]
(or arXiv:2610.21964v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.21964 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 301 cuvinte
Subiecte conexe
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry
arXiv · 11 sept. 2026
Researchers Demonstrate Scalable Quantum Neural Network Training on Quantum Hardware
The Quantum Insider · 9 iun. 2026
Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equation
arXiv · 16 iun. 2026