Stable initialization without the CLT

Abstract:Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support $\mu$P width scaling.

Machine Learning (cs.LG)

arXiv:2609.30633 [cs.LG]

(or arXiv:2609.30633v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.30633 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

assets/gt_bach.wav

cameraman-and-audio.ipynb

moment_matching.py

mup-lr-sweep.ipynb

omega-sweep.ipynb

plot_cameraman_and_audio.py

plot_mup_lr_sweep.py

plot_omega_sweep.py

(4 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.