analiza aiqbench
This study reframes the double descent phenomenon in machine learning through the lens of statistical mechanics, modeling training trajectories as particles diffusing over an energy landscape. By applying the equipartition theorem to parameter degrees of freedom, the authors demonstrate how increasing model size effectively lowers temperature and enhances weight regularization without explicit constraints. This theoretical bridge between physics principles and deep learning dynamics offers a rigorous explanation for why larger models often generalize better despite fitting training data perfectly.
Double descent is the principle of least action
arXiv:2609.19076v1 Announce Type: new Abstract: The test error of a model plotted against its number of parameters $d$ falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature $T$, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the $d$ degrees of freedom in shares of $T/2$, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the $L^2$ norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing $d$, effectively increasing weight regularization.
Abstract:The test error of a model plotted against its number of parameters $d$ falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature $T$, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the $d$ degrees of freedom in shares of $T/2$, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the $L^2$ norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing $d$, effectively increasing weight regularization.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST); Computational Physics (physics.comp-ph); Data Analysis, Statistics and Probability (physics.data-an)
arXiv:2609.19076 [cs.LG]
(or arXiv:2609.19076v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.19076 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 226 cuvinte
Subiecte conexe
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · 23 iul. 2026
Does preservation make sense before we know how to revive?
LessWrong · 15 iun. 2026
Why I Left Google DeepMind
LessWrong · 15 iul. 2026
Podcast: Securing AI Agents: Identity, Authorization, and the DPACT Framework
InfoQ · 21 sept. 2026