aiqbench analysis
This study bridges a critical theoretical gap by establishing the first minimax-optimal generalization rates for deep ReLU networks under the Neural Tangent Kernel regime. The findings confirm that gradient descent methods can achieve performance parity with kernel methods provided network width scales polynomially with depth and sample size. For researchers, this validates the statistical efficiency of standard training dynamics in deep architectures without requiring specialized regularization beyond sufficient overparameterization.
Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks
arXiv:2606.06764v1 Announce Type: cross Abstract: Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.
Abstract:Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.
Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2606.06764 [stat.ML]
(or arXiv:2606.06764v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2606.06764 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 241 words
Related coverage
Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods
arXiv · Jun 8, 2026
Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification
arXiv · Jun 3, 2026
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
arXiv · Aug 26, 2026
Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent
arXiv · Jul 31, 2026