analiza aiqbench
This research identifies a novel "diffusion equilibrium" phase in neural network training where gradient alignment and residual homogeneity significantly enhance generalization, particularly within physics-informed networks. The authors propose a sample-wise re-weighting scheme to leverage this phase transition for faster convergence and improved performance in non-convex optimization landscapes. These findings offer concrete strategies for refining deep learning optimizers by targeting specific saturation-induced compression phenomena during activation.
Learning in PINNs: Phase transition, diffusion equilibrium, and generalization
arXiv:2403.18494v2 Announce Type: replace Abstract: We investigate the learning dynamics of fully-connected neural networks through the lens of the neural gradient signal-to-noise ratio (SNR), examining the behavior of first-order optimizers in non-convex objectives. Interpreting the drift/diffusion phases as proposed in the information bottleneck theory, we identify a third phase termed "diffusion equilibrium" (DE), a stable training phase characterized by highly-ordered neural gradients across the sample space. This phase is marked by an abrupt transition, where sample-wise gradients align (SNR increases), and stable optimizer convergence. Moreover, we find that when homogeneous residuals are also met across the sample space during the DE phase, this leads to better generalization, as the optimization steps are equally sensitive to each sample. Based on this observation, we propose a sample-wise re-weighting scheme, which considerably improves the residual homogeneity and generalization in quadratic loss functions, by targeting the problematic samples with large residuals and vanishing gradients. Finally, we explore the information compression phenomenon, pinpointing a significant saturation-induced compression of activations at the DE phase transition, driven by the sample-wise gradient directional alignment. Interestingly, it is during the saturation of activations that the model converges, with deeper layers experiencing negligible information loss. Supported by experimental examples on physics-informed neural networks (PINNs), which highlight the critical role of gradient agreement due to their inherent PDE-based interdependence of samples, our findings suggest that when both sample-wise gradients and residuals are ordered, this leads to faster convergence and better generalization. Identifying phase transitions could improve deep learning optimization strategies, enhancing physics-informed methods and machine learning performance.
Abstract:We investigate the learning dynamics of fully-connected neural networks through the lens of the neural gradient signal-to-noise ratio (SNR), examining the behavior of first-order optimizers in non-convex objectives. Interpreting the drift/diffusion phases as proposed in the information bottleneck theory, we identify a third phase termed "diffusion equilibrium" (DE), a stable training phase characterized by highly-ordered neural gradients across the sample space. This phase is marked by an abrupt transition, where sample-wise gradients align (SNR increases), and stable optimizer convergence. Moreover, we find that when homogeneous residuals are also met across the sample space during the DE phase, this leads to better generalization, as the optimization steps are equally sensitive to each sample. Based on this observation, we propose a sample-wise re-weighting scheme, which considerably improves the residual homogeneity and generalization in quadratic loss functions, by targeting the problematic samples with large residuals and vanishing gradients. Finally, we explore the information compression phenomenon, pinpointing a significant saturation-induced compression of activations at the DE phase transition, driven by the sample-wise gradient directional alignment. Interestingly, it is during the saturation of activations that the model converges, with deeper layers experiencing negligible information loss. Supported by experimental examples on physics-informed neural networks (PINNs), which highlight the critical role of gradient agreement due to their inherent PDE-based interdependence of samples, our findings suggest that when both sample-wise gradients and residuals are ordered, this leads to faster convergence and better generalization. Identifying phase transitions could improve deep learning optimization strategies, enhancing physics-informed methods and machine learning performance.
Machine Learning (cs.LG)
arXiv:2403.18494 [cs.LG]
(or arXiv:2403.18494v2 [cs.LG] for this version)
Anagnostopoulos, S.J., Toscano, J.D., Stergiopulos, N. and Karniadakis, G.E., 2025. Learning in pinns: Phase transition, diffusion equilibrium, and generalization. Neural Networks, p.107983
https://doi.org/10.1016/j.neunet.2025.107983 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 290 cuvinte
Subiecte conexe
Import AI 473: The US’s superintelligence strategy; human brain in a mouse skull; and machine hermeneutics
Import AI · 21 sept. 2026
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · 8 oct. 2026
Dynamical phase transition in generalized Dicke model with strongly interacting trapped Rydberg ions
arXiv · 7 aug. 2026
How might continual learning affect safety and alignment?
LessWrong · 13 iun. 2026