aiqbench analysis
This research provides a rigorous geometric classification of stationary plateaus in two-layer neural networks, specifically analyzing how neuron splitting operations alter the landscape between local minima and saddle points. By introducing the "inner Hessian" matrix, the study establishes precise conditions under which model width expansion preserves optimization quality or induces degenerate saddle regions. These findings offer critical theoretical insights for understanding the stability and geometry of network reparameterization and scaling strategies.
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
arXiv:2606.04327v1 Announce Type: new Abstract: We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions. We focus on the phenomenon of "neuron splitting" where duplicating a hidden neuron yields an affine set of stationary points in a wider network. We provide a comprehensive classification of all stationary points on these plateaus, determining under what conditions they constitute local minima or saddle points. Our characterization hinges on a per-neuron curvature object we term the "inner Hessian" matrix. Our analysis reveals that the definiteness of the inner Hessian and the choice of splitting coefficients jointly dictate the local geometry of the plateau. We show that "splitting" a local minimum can yield either a mixture of local minima and saddles or an all-saddle plateau, with a concrete sure-saddle region identified under mild assumptions. In contrast, splitting a saddle point always produces a plateau of saddle points. Our results unify and extend prior landscape analyses, elucidating when and how model expansion preserves or alters the nature of stationary points. These findings offer new geometric insights into the effects of width expansion and reparameterization in neural networks.
Abstract:We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions. We focus on the phenomenon of "neuron splitting" where duplicating a hidden neuron yields an affine set of stationary points in a wider network. We provide a comprehensive classification of all stationary points on these plateaus, determining under what conditions they constitute local minima or saddle points. Our characterization hinges on a per-neuron curvature object we term the "inner Hessian" matrix. Our analysis reveals that the definiteness of the inner Hessian and the choice of splitting coefficients jointly dictate the local geometry of the plateau. We show that "splitting" a local minimum can yield either a mixture of local minima and saddles or an all-saddle plateau, with a concrete sure-saddle region identified under mild assumptions. In contrast, splitting a saddle point always produces a plateau of saddle points. Our results unify and extend prior landscape analyses, elucidating when and how model expansion preserves or alters the nature of stationary points. These findings offer new geometric insights into the effects of width expansion and reparameterization in neural networks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
arXiv:2606.04327 [cs.LG]
(or arXiv:2606.04327v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.04327 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 290 words
Related coverage
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
arXiv · Jul 23, 2026
Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks
arXiv · Aug 5, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
A Topological Characterization of Graph Neural Networks via Stochastic Block Model Embeddings on the n-Sphere
arXiv · Jun 9, 2026