aiqbench analysis
This research introduces EnJoi, a diffusion-based data assimilation algorithm that integrates uncertainty estimation from past predictions into the reconstruction of dynamical systems like fluid and traffic flows. By learning the joint distribution of past and future states, the method improves accuracy in scenarios with sparse or non-homogeneous observations, offering a significant advancement for physics-informed machine learning applications.
EnJoi: Ensemble Joint Score Filter for Generative Data Assimilation
arXiv:2609.35944v1 Announce Type: new Abstract: Data Assimilation (DA) aims to recover the full state of a dynamical system that is only partially observed. A solution is to use Score-based models to generate physically consistent trajectories that agree with the observations. These Autoregressive Diffusion models are trained by conditioning on the previous state; however, they do not take into account the uncertainty of their past predictions. We propose a new diffusion-based assimilation algorithm that dynamically balances the confidence in the current state and the new observations. Crucially, we choose to learn the distribution of the joint state containing both the past and future. This allows us to use a modified version of En4DVar, a classical DA algorithm that relies on the covariance of an ensemble of particles. Experiments on fluid and traffic flow simulations show improved reconstruction performance, especially in situations where observations are sparse and non-homogeneous.
Abstract:Data Assimilation (DA) aims to recover the full state of a dynamical system that is only partially observed. A solution is to use Score-based models to generate physically consistent trajectories that agree with the observations. These Autoregressive Diffusion models are trained by conditioning on the previous state; however, they do not take into account the uncertainty of their past predictions. We propose a new diffusion-based assimilation algorithm that dynamically balances the confidence in the current state and the new observations. Crucially, we choose to learn the distribution of the joint state containing both the past and future. This allows us to use a modified version of En4DVar, a classical DA algorithm that relies on the covariance of an ensemble of particles. Experiments on fluid and traffic flow simulations show improved reconstruction performance, especially in situations where observations are sparse and non-homogeneous.
Machine Learning (cs.LG)
arXiv:2609.35944 [cs.LG]
(or arXiv:2609.35944v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.35944 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 235 words
Related coverage
Learning Probabilistic Filters with Strictly Proper Scoring Rules
arXiv · Oct 7, 2026
LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations
arXiv · Oct 6, 2026
Amortized Filtering and Smoothing with Conditional Normalizing Flows
arXiv · Sep 21, 2026
Game-Theoretic Disempowerment: Loss of Control to Agencyless AI
LessWrong · Sep 27, 2026