aiqbench analysis
This research reframes gradient descent in deep ReLU networks as collective dynamics on training-set fields, revealing that information transport across layers relies on a hierarchy of weight-induced Gram operators once depth reaches three. By eliminating explicit weight variables in favor of these operators, the study establishes a closed-form description of residual dynamics that clarifies how deeper architectures manage information flow. The findings offer a theoretical framework for understanding the structural constraints and information bottlenecks inherent in very deep neural networks.
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
arXiv:2606.09744v3 Announce Type: replace Abstract: We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers. Moreover, the conjugate-field dynamics is governed by operators satisfying a backward pullback recursion, of which the weight-induced Gram operators are the first nontrivial instances.
Abstract:We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers. Moreover, the conjugate-field dynamics is governed by operators satisfying a backward pullback recursion, of which the weight-induced Gram operators are the first nontrivial instances.
24 pages. v4: Corrected the hidden-activation dynamics; clarified the concept of field closure. Other minor corrections
Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn)
arXiv:2606.09744 [cs.LG]
(or arXiv:2606.09744v3 [cs.LG] for this version)
Full text under the publisher's open licence · 167 words
Related coverage
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
arXiv · Jun 10, 2026
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
arXiv · Jun 9, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026