aiqbench analysis
This research introduces LEAP, a framework that replaces monolithic LLM reasoning with a modular approach to probabilistic forecasting by separately eliciting likelihoods from individual evidence items. By combining these parameters through an explicit prior and deterministic model, the method improves prediction calibration and preserves reproducible contributions across various agent frameworks. The study demonstrates superior performance in financial and sports forecasting tasks while maintaining efficiency under constrained inference budgets.
LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting
arXiv:2609.01337v1 Announce Type: new Abstract: LLM-based forecasting systems have improved on real-world tasks such as financial markets and sports outcomes, largely through stronger search and tool use. Many systems still ask an LLM to read all collected evidence together and produce the final forecast. We call this design Monolithic Prediction. It can obscure how individual evidence items affect the result and collapse uncertainty across competing outcomes. We propose LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting), which reorganizes how collected evidence is used in the prediction stage. LEAP examines each evidence item separately and elicits likelihood parameters that describe its implications for the target. An explicit prior and a deterministic probabilistic model then combine these likelihoods into a posterior distribution. This procedure supports continuous, single-choice, and multi-choice forecasts while preserving reproducible evidence contributions. We build a benchmark covering forecasting, information-seeking, and browsing tasks, and evaluate LEAP on our own agent loop and several agent CLI frameworks. Given the same evidence, LEAP improves most prediction and calibration metrics across models and remains stronger under controlled comparisons of prior access, inference budget, and aggregation.
Abstract:LLM-based forecasting systems have improved on real-world tasks such as financial markets and sports outcomes, largely through stronger search and tool use. Many systems still ask an LLM to read all collected evidence together and produce the final forecast. We call this design Monolithic Prediction. It can obscure how individual evidence items affect the result and collapse uncertainty across competing outcomes. We propose LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting), which reorganizes how collected evidence is used in the prediction stage. LEAP examines each evidence item separately and elicits likelihood parameters that describe its implications for the target. An explicit prior and a deterministic probabilistic model then combine these likelihoods into a posterior distribution. This procedure supports continuous, single-choice, and multi-choice forecasts while preserving reproducible evidence contributions. We build a benchmark covering forecasting, information-seeking, and browsing tasks, and evaluate LEAP on our own agent loop and several agent CLI frameworks. Given the same evidence, LEAP improves most prediction and calibration metrics across models and remains stronger under controlled comparisons of prior access, inference budget, and aggregation.
Accepted to the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026). 20 pages, 3 figures, and 15 tables. Code: this https URL
Artificial Intelligence (cs.AI)
arXiv:2609.01337 [cs.AI]
(or arXiv:2609.01337v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.01337 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 220 words
Related coverage
Spontaneous verbal evidence synthesis in LLMs
LessWrong · Oct 18, 2026
Article: Artificial Intelligence-Driven Phishing: How Phishing Technique Is Evolving and Implemented
InfoQ · Jun 8, 2026
CoCopula: Conformal Calibration and Copula Aggregation for Reliable Renewable Energy Forecasting
arXiv · Oct 16, 2026
Angles of attack for continual learning safety
LessWrong · Jun 16, 2026