aiqbenchInformații despre AI & calcul cuantic

...

Română · beta. Traducerea a fost făcută automat și este încă în curs de revizuire. Citește pagina în engleză

perspectiva aiqbenchMachine learning

This research challenges the conventional static application of dropout by proposing a depth-varying schedule grounded in mean-field theory near the edge of chaos. The study identifies specific universality classes for different activation functions and demonstrates that concentrating regularization near input layers yields consistent performance gains across MLPs, vision, speech, and financial time series models. While benefits are more pronounced in multi-layer perceptrons than in Transformers, the findings offer a theoretically rigorous approach to optimizing regularization strategies in deep learning architectures.

Sursă care acoperăarXiv

Ce este aceastaLectura unui model asupra articolului și a acoperirii conexe, scrisă pentru aiqbench. Nu sunt cuvintele publicației.

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos

arXiv30 septembrie 20261 min de cititArticolul și perspectiva sunt în engleză.

arXiv:2605.21648v3 Announce Type: replace Abstract: We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.

Abstract:We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.

Accepted at the 43rd International Conference on Machine Learning (ICML 2026). 36 pages, 11 figures

Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)

arXiv:2605.21648 [cs.LG]

(or arXiv:2605.21648v3 [cs.LG] for this version)

Text integral sub licența deschisă a publicației · 133 cuvinte

Citește pe arXiv ↗Întreabă despre acest articol