aiqbench analysis
This research establishes a unified theoretical framework linking online stochastic gradient descent with kernel methods to analyze the generalization error of the last iterate in overparameterized neural networks. By relaxing network-width requirements from exponential to polynomial dependence on sample size, the study bridges the gap between optimization convergence theory and practical statistical learning for streaming data.
Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networks
arXiv:2407.07670v2 Announce Type: replace-cross Abstract: We investigate the generalization performance of online stochastic gradient descent (SGD) for overparameterized two-layer neural networks under the Neural Tangent Kernel (NTK) regime. Leveraging the NTK approximation together with the convergence theory of stochastic approximation in reproducing kernel Hilbert spaces (RKHSs), we develop a unified framework that connects the optimization dynamics of neural networks with kernel-based statistical learning. This framework enables us to derive sharp convergence rates for the \emph{generalization error of the last iterate} of online SGD. The obtained rates coincide with the optimal statistical rates for kernel methods under appropriate source assumptions on target function. In contrast to existing NTK analyses, which primarily establish optimization convergence or analyze averaged SGD, our results directly characterize the statistical behavior of the practically implemented last-iterate online SGD algorithm for streaming data. Moreover, our analysis substantially relaxes the required degree of overparameterization by reducing the network-width requirement from exponential to polynomial dependence on the sample size or the number of optimization iterations. These results provide a unified theoretical perspective on stochastic optimization, kernel methods, and statistical learning for overparameterized neural networks, while significantly narrowing the gap between existing theory and practical deep learning.
Abstract:We investigate the generalization performance of online stochastic gradient descent (SGD) for overparameterized two-layer neural networks under the Neural Tangent Kernel (NTK) regime. Leveraging the NTK approximation together with the convergence theory of stochastic approximation in reproducing kernel Hilbert spaces (RKHSs), we develop a unified framework that connects the optimization dynamics of neural networks with kernel-based statistical learning. This framework enables us to derive sharp convergence rates for the \emph{generalization error of the last iterate} of online SGD. The obtained rates coincide with the optimal statistical rates for kernel methods under appropriate source assumptions on target function. In contrast to existing NTK analyses, which primarily establish optimization convergence or analyze averaged SGD, our results directly characterize the statistical behavior of the practically implemented last-iterate online SGD algorithm for streaming data. Moreover, our analysis substantially relaxes the required degree of overparameterization by reducing the network-width requirement from exponential to polynomial dependence on the sample size or the number of optimization iterations. These results provide a unified theoretical perspective on stochastic optimization, kernel methods, and statistical learning for overparameterized neural networks, while significantly narrowing the gap between existing theory and practical deep learning.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2407.07670 [stat.ML]
(or arXiv:2407.07670v2 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2407.07670 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words
Related coverage
Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models
arXiv · Oct 8, 2026
Stability and Generalization of Stochastic Nonconvex Optimization under Heavy-Tailed Noise: From Gradient Norm to Goldstein Stationarity
arXiv · Oct 6, 2026
Exploding variance of means of exponentials: least-squares to the rescue – Machine Learning Research Blog
francisbach.com · Sep 27, 2026
Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks
arXiv · Jun 8, 2026