analiza aiqbench
This research introduces LEAF, a novel living benchmark designed to rigorously evaluate Large Language Models on event-augmented forecasting by eliminating data contamination and look-ahead leakage. By employing dual-agent cross-validation and recursive retrieval, the study demonstrates that LLMs can effectively extract predictive signals from verified events while significantly reducing information leakage rates. This advancement establishes a critical new standard for validating the true predictive capabilities of AI models in dynamic real-world environments.
LEAF: A Living Benchmark for Event-Augmented Forecasting
arXiv:2605.16358v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remains compromised by pre-training data contamination and look-ahead leakage in automated search. Existing benchmarks either rely on static contexts, restrict evaluations to narrow environments, or fail to audit auxiliary textual events for future information leakage. To establish a rigorous evaluation paradigm, we propose LEAF, the first living benchmark for event-augmented forecasting tasks, including trend, event, and time series forecasting. LEAF couples a recursive retrieval agent system with dual-agent cross-validation to gather comprehensive, relevant, and temporally aligned auxiliary context. A comprehensive audit across 500 tasks by 47 domain specialists demonstrates that our pipeline suppresses future information leakage from 8.6% to 1.6%. Across extensive evaluations of 16 frontier proprietary and open-weight models on our benchmark, we show that LLMs effectively extract signals from verified events to boost trend and event forecasting.
Abstract:Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remains compromised by pre-training data contamination and look-ahead leakage in automated search. Existing benchmarks either rely on static contexts, restrict evaluations to narrow environments, or fail to audit auxiliary textual events for future information leakage. To establish a rigorous evaluation paradigm, we propose LEAF, the first living benchmark for event-augmented forecasting tasks, including trend, event, and time series forecasting. LEAF couples a recursive retrieval agent system with dual-agent cross-validation to gather comprehensive, relevant, and temporally aligned auxiliary context. A comprehensive audit across 500 tasks by 47 domain specialists demonstrates that our pipeline suppresses future information leakage from 8.6% to 1.6%. Across extensive evaluations of 16 frontier proprietary and open-weight models on our benchmark, we show that LLMs effectively extract signals from verified events to boost trend and event forecasting.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.16358 [cs.LG]
(or arXiv:2605.16358v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.16358 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 241 cuvinte
Subiecte conexe
LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
arXiv · 25 aug. 2026
Contrastive Time Series Forecasting with Anomalies
arXiv · 7 oct. 2026
Amazon Research Awards recipients announced
Amazon Science · 27 mai 2026
LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
arXiv · 28 iul. 2026