aiqbench analysis
This study exposes a critical flaw in evaluating machine learning models for wildfire prediction, demonstrating that reliance on Average Precision (AP) can select architectures that significantly over-predict fire spread compared to metrics like F1 or IoU. The findings reveal distinct operational profiles—such as over-predicting versus balanced models—that standard benchmarks fail to distinguish, potentially leading to inefficient resource allocation in real-world containment efforts. For practitioners deploying these systems, the research underscores the necessity of adopting threshold-dependent evaluation metrics to ensure forecasts are operationally viable rather than statistically inflated.
WildfireSpreadBench: The Metric Decides the Model in Wildfire Spread Prediction
arXiv:2609.22191v1 Announce Type: new Abstract: Machine learning is being increasingly used to predict where active wildfires will burn the following day, helping inform evacuation boundaries and containment lines. Most models are evaluated using Average Precision (AP), which summarizes performance across all decision thresholds, although acting on a forecast requires choosing one. We benchmarked five discriminative architectures and one generative model on WildfireSpreadTS using a shared evaluation pipeline and two input configurations. We found that model rankings varied depending on whether performance was measured by AP or by threshold-dependent metrics like F1 and IoU. The highest-AP model flagged 4 to 5 times the area that burned and ranked fifth of six on F1 and IoU, and the most recall-heavy model flagged 16 to 23 times. Models with more usable predictions had AP scores 24 to 37 lower. Across architectures, we identified three distinct prediction profiles: over-predicting, balanced, and under-predicting, which AP alone could not distinguish. Expanding the input from 7 to 23 channels changed AP by 0.03 on average, against a 0.21 to 0.24 spread across architectures. These results show AP alone can favor models whose predictions are poorly suited for operational wildfire forecasting.
Abstract:Machine learning is being increasingly used to predict where active wildfires will burn the following day, helping inform evacuation boundaries and containment lines. Most models are evaluated using Average Precision (AP), which summarizes performance across all decision thresholds, although acting on a forecast requires choosing one. We benchmarked five discriminative architectures and one generative model on WildfireSpreadTS using a shared evaluation pipeline and two input configurations. We found that model rankings varied depending on whether performance was measured by AP or by threshold-dependent metrics like F1 and IoU. The highest-AP model flagged 4 to 5 times the area that burned and ranked fifth of six on F1 and IoU, and the most recall-heavy model flagged 16 to 23 times. Models with more usable predictions had AP scores 24 to 37 lower. Across architectures, we identified three distinct prediction profiles: over-predicting, balanced, and under-predicting, which AP alone could not distinguish. Expanding the input from 7 to 23 channels changed AP by 0.03 on average, against a 0.21 to 0.24 spread across architectures. These results show AP alone can favor models whose predictions are poorly suited for operational wildfire forecasting.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2609.22191 [cs.LG]
(or arXiv:2609.22191v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.22191 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 286 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
AXRP Episode 50 - Eli Lifland on AI 2027
LessWrong · Aug 3, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026