Residual-based attention in physics-informed neural networks

arXiv:2307.00379v2 Announce Type: replace Abstract: Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs that accelerates the convergence of dynamic or static systems. This simple yet effective attention mechanism is a bounded function of the evolving cumulative residuals and aims to make the optimizer aware of problematic regions at no extra computational cost or adversarial learning. We illustrate that this general method consistently achieves one order of magnitude faster convergence than vanilla PINNs and a minimum relative $L^{2}$ error of $\mathcal{O}(10^{-5})$, on typical benchmarks of the literature. The method is further tested on the inverse solution of the Navier-Stokes within the brain perivascular spaces, where it considerably improves the prediction accuracy. Furthermore, an ablation study is performed for each case to identify the contribution of the components that enhance the vanilla PINN formulation. Evident from the convergence trajectories is the ability of the optimizer to effectively escape from poor local minima or saddle points while focusing on the challenging domain regions, which consistently have a high residual score. We believe that alongside exact boundary conditions and other model reparameterizations, this type of attention mask could be an essential element for fast training of both PINNs and neural operators.

Abstract:Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs that accelerates the convergence of dynamic or static systems. This simple yet effective attention mechanism is a bounded function of the evolving cumulative residuals and aims to make the optimizer aware of problematic regions at no extra computational cost or adversarial learning. We illustrate that this general method consistently achieves one order of magnitude faster convergence than vanilla PINNs and a minimum relative $L^{2}$ error of $\mathcal{O}(10^{-5})$, on typical benchmarks of the literature. The method is further tested on the inverse solution of the Navier-Stokes within the brain perivascular spaces, where it considerably improves the prediction accuracy. Furthermore, an ablation study is performed for each case to identify the contribution of the components that enhance the vanilla PINN formulation. Evident from the convergence trajectories is the ability of the optimizer to effectively escape from poor local minima or saddle points while focusing on the challenging domain regions, which consistently have a high residual score. We believe that alongside exact boundary conditions and other model reparameterizations, this type of attention mask could be an essential element for fast training of both PINNs and neural operators.

Machine Learning (cs.LG); Computational Physics (physics.comp-ph)

arXiv:2307.00379 [cs.LG]

(or arXiv:2307.00379v2 [cs.LG] for this version)

Anagnostopoulos, S.J., Toscano, J.D., Stergiopulos, N. and Karniadakis, G.E., 2024. Residual-based attention in physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 421, p.116805

https://doi.org/10.1016/j.cma.2024.116805 DOI(s) linking to related resources

aiqbench analysis

Residual-based attention in physics-informed neural networks

Related coverage