aiqbench takeMachine Learning

This research challenges the conventional static application of dropout by proposing a depth-varying schedule grounded in mean-field theory near the edge of chaos. The study identifies specific universality classes for different activation functions and demonstrates that concentrating regularization near input layers yields consistent performance gains across MLPs, vision, speech, and financial time series models. While benefits are more pronounced in multi-layer perceptrons than in Transformers, the findings offer a theoretically rigorous approach to optimizing regularization strategies in deep learning architectures.

Source coveringarXiv

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos

arXivSeptember 30, 20261 min read

arXiv:2605.21648v3 Announce Type: replace Abstract: We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.

Abstract:We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.

Accepted at the 43rd International Conference on Machine Learning (ICML 2026). 36 pages, 11 figures

Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)

arXiv:2605.21648 [cs.LG]

(or arXiv:2605.21648v3 [cs.LG] for this version)

Full text under the publisher's open licence · 133 words

Read at arXiv ↗Ask about this article