aiqbench analysis
This arXiv paper introduces FABench, a benchmark that backtests AI forecasting agents on past events, which avoids the wait for live resolutions and the staleness of fixed corpora. The reported results are that retrieval improves Brier scores for all eight evaluated models, and the strongest agents roughly match prediction-market consensus on a 54-question subset. The authors also note recurring failures in how agents weigh evidence, so readers should treat these as early benchmark results rather than proof of reliable forecasting.
Benchmarking Forecasting Agents on Yesterday's Tomorrows
arXiv:2610.40940v1 Announce Type: cross Abstract: Digital agents can now gather and organize information, reason over it, and use tools to complete complex tasks. Forecasting puts all of these abilities to the test: it requires integrating incomplete evidence, inferring causal drivers, and committing to calibrated probabilistic judgments. Existing benchmarks are poorly suited to measuring and improving this ability. Live benchmarks provide feedback only after events resolve, so every evaluation cycle waits on the calendar; fixed-corpus evaluations return results immediately but confine research to pre-collected documents. We introduce FABench, a scalable backtesting environment for Forecasting Agents. An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs; its first release poses 287 questions across nine domains. Agents research each question freely in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage. Evaluating agents built on eight recent models, we find that retrieval improves the Brier score of all eight in every retrieval setting and lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it. Research traces reveal recurring failures in how agents weigh and apply evidence, leaving ample room for improvement. We release our code, data, and archive environment as a foundation for community research on forecasting agents.
Abstract:Digital agents can now gather and organize information, reason over it, and use tools to complete complex tasks. Forecasting puts all of these abilities to the test: it requires integrating incomplete evidence, inferring causal drivers, and committing to calibrated probabilistic judgments. Existing benchmarks are poorly suited to measuring and improving this ability. Live benchmarks provide feedback only after events resolve, so every evaluation cycle waits on the calendar; fixed-corpus evaluations return results immediately but confine research to pre-collected documents. We introduce FABench, a scalable backtesting environment for Forecasting Agents. An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs; its first release poses 287 questions across nine domains. Agents research each question freely in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage. Evaluating agents built on eight recent models, we find that retrieval improves the Brier score of all eight in every retrieval setting and lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it. Research traces reveal recurring failures in how agents weigh and apply evidence, leaving ample room for improvement. We release our code, data, and archive environment as a foundation for community research on forecasting agents.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2610.40940 [cs.AI]
(or arXiv:2610.40940v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.40940 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 326 words
Related coverage
FinVerse: Financial Time-Series Benchmark
arXiv · Aug 5, 2026
Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance
arXiv · Oct 8, 2026
FETS Benchmark: Foundation Models Enable Scalable and Generalizable Energy Time Series Forecasting
arXiv · Jul 21, 2026
WPBench: A Comprehensive Benchmark for Wind Power Forecasting
arXiv · Sep 22, 2026