aiqbench analysis
This research investigates how the geometric structure of data shortcuts—specifically whether they are collapsed or dispersed—affects the generalization capabilities of spectral gradient descent versus standard gradient descent in overparameterized matrix classification. The study reveals that minor changes in shortcut geometry can reverse which optimization method yields better generalization, highlighting a critical dependency on matrix geometry for implicit bias. Furthermore, it demonstrates that continued training beyond a single step can lead to convergence toward directions with substantially worse generalization performance.
Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias
arXiv:2609.32270v1 Announce Type: cross Abstract: We study the generalization of spectral gradient descent (SpecGD) in overparameterized matrix classification with corrupted labels. Each input combines a shared low-rank signal with a rank-one sample-specific perturbation, referred to as a shortcut, that enables memorization but does not generalize. We contrast collapsed shortcuts, which share a singular direction, with dispersed shortcuts, which occupy distinct singular directions. Changing only this geometry can reverse the relative generalization of GD and SpecGD: collapsed shortcuts can favor SpecGD, while dispersed shortcuts can favor GD. In the dispersed regime, exact shortcut orthogonality eliminates the signal from the late-stage SpecGD direction, while vanishing random correlations collectively generate a small but generalization-relevant signal through a second-order effect. To identify the direction selected by SpecGD, which the spectral max-margin problem alone does not determine, we combine a refined analysis of its dual with the exponentiated-gradient dynamics of normalized loss weights. Finally, we show that a single SpecGD step can already interpolate and generalize well, while continued training converges to a direction with substantially worse generalization.
Abstract:We study the generalization of spectral gradient descent (SpecGD) in overparameterized matrix classification with corrupted labels. Each input combines a shared low-rank signal with a rank-one sample-specific perturbation, referred to as a shortcut, that enables memorization but does not generalize. We contrast collapsed shortcuts, which share a singular direction, with dispersed shortcuts, which occupy distinct singular directions. Changing only this geometry can reverse the relative generalization of GD and SpecGD: collapsed shortcuts can favor SpecGD, while dispersed shortcuts can favor GD. In the dispersed regime, exact shortcut orthogonality eliminates the signal from the late-stage SpecGD direction, while vanishing random correlations collectively generate a small but generalization-relevant signal through a second-order effect. To identify the direction selected by SpecGD, which the spectral max-margin problem alone does not determine, we combine a refined analysis of its dual with the exponentiated-gradient dynamics of normalized loss weights. Finally, we show that a single SpecGD step can already interpolate and generalize well, while continued training converges to a direction with substantially worse generalization.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2609.32270 [stat.ML]
(or arXiv:2609.32270v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2609.32270 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 265 words
Related coverage
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization
arXiv · Oct 6, 2026
Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates
arXiv · Oct 12, 2026
Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
arXiv · Oct 17, 2026