aiqbench analysis
This research proposes a novel polynomial activation function that enables deep networks to train without normalization layers, bridging theoretical loss surface analysis with practical model design. By solving the associated polynomial systems exactly at small scales, the authors derive specific prescriptions for initializing and training large-scale stacks on standard benchmarks like MNIST. The work offers significant insights into network architecture and optimization dynamics, directly advancing the understanding of deep learning mechanics.
Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks
arXiv:2610.20069v1 Announce Type: new Abstract: What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + {\lambda}z 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.
Abstract:What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + {\lambda}z 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.20069 [cs.LG]
(or arXiv:2610.20069v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20069 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 189 words
Related coverage
Challenge: Hand coding weights for efficient sequence memorisation
AI Alignment Forum · Jul 23, 2026
Promise of Graph Sparsification and Decomposition for Noise Reduction in QAOA: Analysis for Trapped-Ion Compilations
Quantum Journal · Aug 7, 2026
How to Size GPUs for AI Inference and TCO Without Overspending
NVIDIA · Sep 1, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026