aiqbench analysis
This research offers a novel metric for detecting subspace stabilization in the Hessian matrix during neural network training, independent of parameter change magnitude. The authors further demonstrate that the gradient covariance matrix exhibits similar stabilization properties, providing an explicit approximation for its top subspaces. These findings deepen the theoretical understanding of second-order optimization dynamics and eigenvalue separation in deep learning.
Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
arXiv:2609.31983v1 Announce Type: new Abstract: The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-$(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.
Abstract:The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-$(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.
Machine Learning (cs.LG); Optimization and Control (math.OC)
arXiv:2609.31983 [cs.LG]
(or arXiv:2609.31983v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.31983 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 293 words
Related coverage
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
arXiv · Jun 3, 2026
Some Interesting Papers on RLVR
LessWrong · Jun 9, 2026
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
arXiv · Jun 5, 2026
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
arXiv · Jun 3, 2026