Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

Abstract:We study feed-forward ReLU networks with fixed readout and quadratic loss. The aim is to rewrite gradient descent not primarily as a dynamics in weight space, but as a collective dynamics closed in terms of fields defined on the training-set space. For a single hidden layer, the weight variables can be eliminated from the activation dynamics, yielding a closed equation for the residuals governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. For deeper networks, the residual dynamics retains a clean layer-wise kernel structure. However, from depth three onward, closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers. Moreover, the conjugate-field dynamics is governed by operators satisfying a backward pullback recursion, of which the weight-induced Gram operators are the first nontrivial instances.

26 pages. v3: Added Proposition 4 on recursive transport closure of the conjugate-field dynamics. Minor clarifications

Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn)

arXiv:2606.09744 [cs.LG]

(or arXiv:2606.09744v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2606.09744 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.