analiza aiqbench
This research models Stochastic Gradient Descent dynamics as a percolation process, revealing how architectural symmetries cause subnetworks to merge in discrete blocks rather than sequentially. The study identifies variance spikes as signatures of these structural transitions and demonstrates that this trapping mechanism extends to Adam and AdamW optimizers under heavy-tailed noise conditions. By linking optimization trajectories to physical phase transitions, the work offers a theoretical framework for understanding convergence behavior in deep learning.
Percolation Dynamics in Optimization: Variance Cascades and Discrete Scale Invariance
arXiv:2609.02373v1 Announce Type: new Abstract: We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.
Abstract:We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.
Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Artificial Intelligence (cs.AI)
arXiv:2609.02373 [cs.LG]
(or arXiv:2609.02373v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.02373 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 134 cuvinte
Subiecte conexe
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
arXiv · 9 sept. 2026
Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization
Meta Engineering · 15 iul. 2026
Performance Profiling on AMD GPUs – Part 5: Profiling-Driven Kernel Optimization with an AI Code-Assist Tool
AMD · 16 iul. 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · 15 aug. 2026