aiqbench analysis
This paper tests whether LSTM ensemble spread, a common uncertainty signal in deep-learning streamflow forecasting, stays reliable at high flows and beyond the training data. Spread consistently understates error, by up to 18x for extreme flows outside the training record, even though its error-ranking score is near-oracle. For ML practitioners, the takeaway is that good uncertainty ranking does not imply calibrated magnitudes, and that distribution shift is most dangerous exactly where risk matters most.
Ranked Right, Sized Wrong: Ensemble Spread Fails Silently at Streamflow Extremes
arXiv:2610.20947v1 Announce Type: cross Abstract: Deep learning models, particularly Long Short-Term Memory networks, are increasingly used for operational streamflow forecasting, with ensemble spread providing a signal of predictive uncertainty. As climate change increases the frequency of hydroclimatic extremes, these models will increasingly operate under flow conditions that are rare in their historical records. We therefore evaluate whether ensemble spread remains informative under increasing flows, examining both its magnitude relative to error and its ability to rank errors. Prediction error is approximately $2\times$ the corrected spread at ordinary flows and up to $4\times$ at the highest flows, with the discrepancy increasing with flow (log-log slope $1.30$, 95% CI: $[1.20,1.41]$). Under-dispersion is strongest in observed extreme flows beyond the training record, reaching an $18\times$ discrepancy. This is pervasive and not explainable by conditional mean bias. Area Under the Sparsification Error, a standard distribution-free metric of uncertainty ranking, describing how well a model knows when it is uncertain, indicates near-oracle ranking for ensemble spread ($0.071$), yet ordering by predicted discharge alone nearly matches it. Informative uncertainty ranking can coexist with severe under-dispersion. In deployment, error ranking cannot be verified, yet the spread is available, and it is relied on as the uncertainty signal under increasingly extreme conditions, while systematically understating risk, most severely for the extreme events that matter most.
Abstract:Deep learning models, particularly Long Short-Term Memory networks, are increasingly used for operational streamflow forecasting, with ensemble spread providing a signal of predictive uncertainty. As climate change increases the frequency of hydroclimatic extremes, these models will increasingly operate under flow conditions that are rare in their historical records. We therefore evaluate whether ensemble spread remains informative under increasing flows, examining both its magnitude relative to error and its ability to rank errors. Prediction error is approximately $2\times$ the corrected spread at ordinary flows and up to $4\times$ at the highest flows, with the discrepancy increasing with flow (log-log slope $1.30$, 95% CI: $[1.20,1.41]$). Under-dispersion is strongest in observed extreme flows beyond the training record, reaching an $18\times$ discrepancy. This is pervasive and not explainable by conditional mean bias. Area Under the Sparsification Error, a standard distribution-free metric of uncertainty ranking, describing how well a model knows when it is uncertain, indicates near-oracle ranking for ensemble spread ($0.071$), yet ordering by predicted discharge alone nearly matches it. Informative uncertainty ranking can coexist with severe under-dispersion. In deployment, error ranking cannot be verified, yet the spread is available, and it is relied on as the uncertainty signal under increasingly extreme conditions, while systematically understating risk, most severely for the extreme events that matter most.
Accepted at the Tackling Climate Change with Machine Learning workshop at NeurIPS 2026. 13 pages, 2 figures, 6 tables. Code: this https URL
Machine Learning (stat.ML); Machine Learning (cs.LG); Geophysics (physics.geo-ph)
arXiv:2610.20947 [stat.ML]
(or arXiv:2610.20947v1 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2610.20947 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 334 words
Related coverage
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · Aug 15, 2026
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · Oct 8, 2026
Iteratively Tuning hipBLASLt TensileLite Kernels: A Smaller Search, a Faster Kernel
AMD · Sep 9, 2026