Fractal basin geometry and the limits to predictability and reproducibility in deep learning

arXiv:2510.05606v2 Announce Type: replace Abstract: Fundamental limits to predictability are central to our understanding of many physical and computational systems. In deep learning, training outcomes are notoriously difficult to predict and reproduce, yet the mechanism underlying these difficulties has remained unclear. Here we propose fractal, riddled basin geometry as a unifying dynamical-systems mechanism: every initialization leading to one solution lies arbitrarily close to initializations leading to different solutions, forming a fat fractal with structure at arbitrarily fine scales. Using a minimal neural network model, we analytically establish the classical route to riddling and derive a near-zero uncertainty exponent: even sharp increases in initialization precision yield only marginal gains in outcome predictability. In deep networks, intertwined training destinations and near-zero parameter-space exponents provide robust signatures consistent with riddling across tested architectures, optimizers and training protocols, offering a common dynamical account of otherwise disparate empirical observations. We further introduce functional uncertainty exponents, which extend these diagnostics of scale-insensitive irreproducibility from parameter space to the learned functions themselves. Together, our results support riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning, with practical implications for optimization and the reliable deployment of artificial intelligence.

Abstract:Fundamental limits to predictability are central to our understanding of many physical and computational systems. In deep learning, training outcomes are notoriously difficult to predict and reproduce, yet the mechanism underlying these difficulties has remained unclear. Here we propose fractal, riddled basin geometry as a unifying dynamical-systems mechanism: every initialization leading to one solution lies arbitrarily close to initializations leading to different solutions, forming a fat fractal with structure at arbitrarily fine scales. Using a minimal neural network model, we analytically establish the classical route to riddling and derive a near-zero uncertainty exponent: even sharp increases in initialization precision yield only marginal gains in outcome predictability. In deep networks, intertwined training destinations and near-zero parameter-space exponents provide robust signatures consistent with riddling across tested architectures, optimizers and training protocols, offering a common dynamical account of otherwise disparate empirical observations. We further introduce functional uncertainty exponents, which extend these diagnostics of scale-insensitive irreproducibility from parameter space to the learned functions themselves. Together, our results support riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning, with practical implications for optimization and the reliable deployment of artificial intelligence.

Machine Learning (cs.LG)

arXiv:2510.05606 [cs.LG]

(or arXiv:2510.05606v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2510.05606 Focus to learn more arXiv-issued DOI via DataCite

Supplementary_Information.pdf

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Fractal basin geometry and the limits to predictability and reproducibility in deep learning

Related coverage