aiqbench analysis
This study critically evaluates the efficacy of physics-informed pre-training using PDE residuals for scientific foundation models, revealing significant limitations when applied to complex systems with discontinuities or novel boundary conditions. The findings suggest that while residual-based methods offer benefits in idealized settings, achieving broad transferability likely requires more principled integration of physical knowledge directly into model architectures rather than relying solely on data-driven approaches.
Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models
arXiv:2503.19081v2 Announce Type: replace Abstract: Scientific foundation models (SciFMs) aim to learn generalizable representations of physical systems governed by partial differential equations (PDEs), enabling transfer across tasks and domains. While physics-informed methods, which leverage PDE residuals as supervisory signals, have shown promise in scientific machine learning (SciML) for improving accuracy and reducing data requirements, their potential in the context of SciFMs remains relatively unexplored. In this evaluation study, we investigate whether (and how) physics-informed pre-training improves the generalization, robustness, and data efficiency of SciFMs. We conduct systematic experiments across a diverse set of PDEs, ranging from simple problems with periodic boundary conditions to more challenging systems such as the Navier-Stokes equations and non-periodic geometries. Our results show that physics-informed pre-training provides clear benefits in ``nice,'' e.g., structured, well-aligned settings: it enhances generalization and reduces data dependence, compared to data-only pre-training. However, these advantages diminish significantly as the downstream tasks become ``harder,'' e.g., as they involve discontinuities or deviate from the pre-training distribution. In complex or structurally different problems, such as those involving new boundary conditions or PDE operators, physics-informed models may perform only on par with---or even worse---than data-driven baselines. While residual-based pre-training helps in idealized regimes, realizing broadly transferable SciFMs will likely require subtler spatiotemporal inductive biases and more principled integration of physical knowledge into model architectures.
Abstract:Scientific foundation models (SciFMs) aim to learn generalizable representations of physical systems governed by partial differential equations (PDEs), enabling transfer across tasks and domains. While physics-informed methods, which leverage PDE residuals as supervisory signals, have shown promise in scientific machine learning (SciML) for improving accuracy and reducing data requirements, their potential in the context of SciFMs remains relatively unexplored. In this evaluation study, we investigate whether (and how) physics-informed pre-training improves the generalization, robustness, and data efficiency of SciFMs. We conduct systematic experiments across a diverse set of PDEs, ranging from simple problems with periodic boundary conditions to more challenging systems such as the Navier-Stokes equations and non-periodic geometries. Our results show that physics-informed pre-training provides clear benefits in ``nice,'' e.g., structured, well-aligned settings: it enhances generalization and reduces data dependence, compared to data-only pre-training. However, these advantages diminish significantly as the downstream tasks become ``harder,'' e.g., as they involve discontinuities or deviate from the pre-training distribution. In complex or structurally different problems, such as those involving new boundary conditions or PDE operators, physics-informed models may perform only on par with---or even worse---than data-driven baselines. While residual-based pre-training helps in idealized regimes, realizing broadly transferable SciFMs will likely require subtler spatiotemporal inductive biases and more principled integration of physical knowledge into model architectures.
Machine Learning (cs.LG)
arXiv:2503.19081 [cs.LG]
(or arXiv:2503.19081v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2503.19081 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 305 words
