aiqbench analysis
This research provides a rigorous mathematical framework for quantifying the affine geometry of ReLU networks using conditional Kac-Rice formulas, bridging probabilistic theory with network structure analysis. By deriving exact expressions for activation switches and scalar kinks, the study reveals that supervised training redistributes rather than simply contracts this intrinsic complexity. These findings offer deep theoretical insights into how neural architectures evolve from random initialization to trained states, which is critical for advancing our understanding of machine learning dynamics.
Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas
arXiv:2609.32695v1 Announce Type: cross Abstract: We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.
Abstract:We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.
Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR)
arXiv:2609.32695 [stat.ML]
(or arXiv:2609.32695v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2609.32695 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 297 words
Related coverage
Exploding variance of means of exponentials: least-squares to the rescue – Machine Learning Research Blog
francisbach.com · Sep 27, 2026
Second Maximum of a Gaussian Random Field and Exact (t-)Spacing test
arXiv · Oct 16, 2026
Diffusion Operator Geometry of Feedforward Representations
arXiv · Aug 7, 2026
Population loss in shallow ReLU networks: Bias & families of critical points
arXiv · Sep 28, 2026