aiqbench analysis
This paper bridges the gap between practical deep learning success and theoretical optimization by introducing a generalized framework for convexity and smoothness using Legendre functions. It establishes rigorous convergence rates for SGD variants on diverse architectures like Transformers and CNNs, demonstrating that optimal training dynamics hinge on controlling gradient energy and structure matrix eigenvalues rather than strict convexity assumptions.
Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training
arXiv:2608.09523v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory does not fully explain this success: conventional analyses rest on differentiability, convexity, and smoothness, assumptions that DNN objectives routinely violate. We close this gap by generalizing convexity and smoothness, through Legendre functions and convex conjugation, into a unified optimization framework for DNN training. We introduce \(\mathcal{H}(\psi)\)-convexity and \(\mathcal{H}(\Psi)\)-smoothness, a single formalism that encompasses convex and non-convex as well as smooth and non-smooth objectives and reveals a natural duality between generalized convexity and smoothness. Building on these notions, we derive generalized gradient descent (GD) and generalized SGD via convex conjugation. We prove that generalized GD attains an optimal learning rate of exactly \(1\), and we establish rigorous gradient-energy-based convergence rates for both optimizers. We further recast DNN training as a composite optimization problem and show that its convergence hinges on jointly reducing the gradient energy and controlling the extreme eigenvalues of the structure matrix; a model capacity risk quantifies the residual error that no optimizer can remove. Experiments on six datasets (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, TREC, and SST-2) and on convolutional, residual, and Transformer architectures, with five loss functions, are consistent with the predicted training dynamics.
Abstract:Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory does not fully explain this success: conventional analyses rest on differentiability, convexity, and smoothness, assumptions that DNN objectives routinely violate. We close this gap by generalizing convexity and smoothness, through Legendre functions and convex conjugation, into a unified optimization framework for DNN training. We introduce \(\mathcal{H}(\psi)\)-convexity and \(\mathcal{H}(\Psi)\)-smoothness, a single formalism that encompasses convex and non-convex as well as smooth and non-smooth objectives and reveals a natural duality between generalized convexity and smoothness. Building on these notions, we derive generalized gradient descent (GD) and generalized SGD via convex conjugation. We prove that generalized GD attains an optimal learning rate of exactly \(1\), and we establish rigorous gradient-energy-based convergence rates for both optimizers. We further recast DNN training as a composite optimization problem and show that its convergence hinges on jointly reducing the gradient energy and controlling the extreme eigenvalues of the structure matrix; a model capacity risk quantifies the residual error that no optimizer can remove. Experiments on six datasets (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, TREC, and SST-2) and on convolutional, residual, and Transformer architectures, with five loss functions, are consistent with the predicted training dynamics.
Machine Learning (cs.LG)
arXiv:2608.09523 [cs.LG]
(or arXiv:2608.09523v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2608.09523 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 296 words
Related coverage
When bureaucracy gives you something it shouldn't
LessWrong · Oct 14, 2026
Generalized Score Matching for Parameter Estimation on Convex Domains
arXiv · Sep 11, 2026
Predicting Alignment Generalization with Value Representations
arXiv · Oct 10, 2026
T-TAMER: Provably Taming Trade-offs in ML Serving
arXiv · Aug 4, 2026