aiqbench analysis
This research challenges the conventional view that flat minima in neural network loss landscapes are a primary driver of generalization, arguing instead that curvature is merely an artifact of parameter scaling rather than a functional property. The author introduces a function-level metric based on adaptability and freedom within learned constraints to better predict held-out accuracy across diverse networks. This work offers significant theoretical insight for researchers investigating the fundamental mechanisms behind deep learning robustness without relying on misleading geometric interpretations.
Are Flat Minima an Illusion?
arXiv:2605.05209v2 Announce Type: replace-cross Abstract: Flat minima are an account of why deep networks generalise. However flatness is a matter of form (parameters), while generalisation is of function. The same function can be a result of many different parameterisations. I demonstrate this by rescaling ReLU networks, changing raw Hessian trace by up to $99$ times while every prediction remains fixed. Raw curvature cannot identify a function-level explanation. Previous theoretical work traced generalisation to the weakness of constraints implied by function, meaning the freedom a model retains within the bounds of what it has learned to be correct. A policy is weaker when more future commitments remain compatible with what it has learned, allowing more freedom to adapt. To measure this for neural networks, I freeze the last hidden representation and ask whether each of 512 sampled label bundles can be met by a replacement affine classifier. The resulting joint completion score is invariant under invertible linear mixing and translation of feature coordinates. Across two predeclared cohorts of 100 networks, it predicts held-out accuracy with rank correlations $0.29$ and $0.47$. Raw Hessian trace and relative flatness have no multiplicity-corrected association. To put it provocatively, freedom is correlated with adaptability, while flatness is a matter of description.
Abstract:Flat minima are an account of why deep networks generalise. However flatness is a matter of form (parameters), while generalisation is of function. The same function can be a result of many different parameterisations. I demonstrate this by rescaling ReLU networks, changing raw Hessian trace by up to $99$ times while every prediction remains fixed. Raw curvature cannot identify a function-level explanation. Previous theoretical work traced generalisation to the weakness of constraints implied by function, meaning the freedom a model retains within the bounds of what it has learned to be correct. A policy is weaker when more future commitments remain compatible with what it has learned, allowing more freedom to adapt. To measure this for neural networks, I freeze the last hidden representation and ask whether each of 512 sampled label bundles can be met by a replacement affine classifier. The resulting joint completion score is invariant under invertible linear mixing and translation of feature coordinates. Across two predeclared cohorts of 100 networks, it predicts held-out accuracy with rank correlations $0.29$ and $0.47$. Raw Hessian trace and relative flatness have no multiplicity-corrected association. To put it provocatively, freedom is correlated with adaptability, while flatness is a matter of description.
27 pages, 1 figure. Major revision adds an affine-invariant joint completion score, PAC-Bayes certificates, a task-alignment theorem, three predeclared 100-network cohorts, a random-label control, and expanded references. Submitted to JMLR
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.05209 [cs.LG]
(or arXiv:2605.05209v2 [cs.LG] for this version)
Full text under the publisher's open licence · 244 words
Related coverage
A Function-Centric Perspective on Flat and Sharp Minima
arXiv · Oct 15, 2026
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
arXiv · Sep 9, 2026
Scaling Limits of Constant-Stepsize SGD at Flat Minima
arXiv · Jul 21, 2026
Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
arXiv · Jun 16, 2026