analiza aiqbench
This research introduces a novel framework using Feynman diagrams to efficiently compute finite-width corrections for Neural Tangent Kernels (NTKs), bridging the gap between infinite-width theoretical limits and practical deep network training dynamics. By simplifying complex algebraic manipulations, this method enables precise predictions of layer-wise recursion relations and stability properties in networks with widths greater than 20. The work significantly advances our analytical understanding of feature learning and NTK evolution beyond traditional Gaussian approximations.
Finite-Width Neural Tangent Kernels from Feynman Diagrams
arXiv:2508.11522v4 Announce Type: replace Abstract: Neural tangent kernels (NTKs) are a powerful tool for analyzing deep, non-linear neural networks. In the infinite-width limit, NTKs can easily be computed for most common architectures, yielding full analytic control over the training dynamics. However, at infinite width, important properties of training such as NTK evolution or feature learning are absent. Nevertheless, finite width effects can be included by computing corrections to the Gaussian statistics at infinite width. We introduce Feynman diagrams for computing finite-width corrections to NTK statistics. These dramatically simplify the necessary algebraic manipulations and enable the computation of layer-wise recursion relations for arbitrary statistics involving preactivations, NTKs and certain higher-derivative tensors (dNTK and ddNTK) required to predict the training dynamics at leading order. We demonstrate the feasibility of our framework by extending stability results for deep networks from preactivations to NTKs and proving the absence of finite-width corrections for scale-invariant nonlinearities such as ReLU on the diagonal of the Gram matrix of the NTK. We numerically implement the complete set of equations necessary to compute the first-order corrections for arbitrary inputs and demonstrate that the results follow the statistics of sampled neural networks for widths $n\gtrsim 20$.
Abstract:Neural tangent kernels (NTKs) are a powerful tool for analyzing deep, non-linear neural networks. In the infinite-width limit, NTKs can easily be computed for most common architectures, yielding full analytic control over the training dynamics. However, at infinite width, important properties of training such as NTK evolution or feature learning are absent. Nevertheless, finite width effects can be included by computing corrections to the Gaussian statistics at infinite width. We introduce Feynman diagrams for computing finite-width corrections to NTK statistics. These dramatically simplify the necessary algebraic manipulations and enable the computation of layer-wise recursion relations for arbitrary statistics involving preactivations, NTKs and certain higher-derivative tensors (dNTK and ddNTK) required to predict the training dynamics at leading order. We demonstrate the feasibility of our framework by extending stability results for deep networks from preactivations to NTKs and proving the absence of finite-width corrections for scale-invariant nonlinearities such as ReLU on the diagonal of the Gram matrix of the NTK. We numerically implement the complete set of equations necessary to compute the first-order corrections for arbitrary inputs and demonstrate that the results follow the statistics of sampled neural networks for widths $n\gtrsim 20$.
Machine Learning (cs.LG); High Energy Physics - Theory (hep-th)
arXiv:2508.11522 [cs.LG]
(or arXiv:2508.11522v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 208 cuvinte
Subiecte conexe
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent
arXiv · 31 iul. 2026
Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
arXiv · 6 oct. 2026
Resolution-Independent Analysis of Encoder--Decoder Operator Learning via Limiting Kernels
arXiv · 15 sept. 2026