aiqbench analysis
This paper is a theoretical study of gradient descent on linearly separable data, showing that delayed generalization, where training accuracy saturates long before test accuracy improves, can arise even in a linear model with an exponential-tail loss. The authors split training into population, support-vector-dominated unlearning, and generalization phases, and tie the delay to a finite-time timescale in the hyperplane dynamics. For ML readers, the value is a simpler, analyzable model of a phenomenon usually attributed to deep architectures, though validation beyond controlled linear settings and a single MNIST check remains limited.
On the Dynamics of Delayed Generalization in Linear Models
arXiv:2602.08302v2 Announce Type: replace Abstract: Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.
Abstract:Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.08302 [cs.LG]
(or arXiv:2602.08302v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.08302 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 265 words
Related coverage
A Mechanistic Analysis of Transformers for Dynamical Systems
arXiv · Aug 6, 2026
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
arXiv · Sep 16, 2026
Bilinear Mamba-Koopman Neural MPC for Varying Dynamics
arXiv · Oct 14, 2026
Qubit transfer errors fall to a quadratic level with STIRAP
Quantum Zeitgeist · Oct 13, 2026