OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking

Abstract:Forecasting is a composite capability encompassing information gathering, evidence integration, situational assessment, and decision-making. Large language models (LLMs) combine reasoning and tool use, offering new avenues for automated forecasting and decision support. Yet LLM forecasting evaluation faces a structural challenge: live benchmarks support prospective assessment, but limited question reuse and changing information conditions make results difficult to reproduce and compare; retrospective benchmarks offer reproducibility but risk information leakage. We introduce OracleProto, a reproducible framework for evaluating LLM forecasting in a controlled, temporally masked environment. Through model knowledge cutoff constraints and temporal masking, OracleProto constructs a temporal sandbox for real-world web retrieval, audits information boundaries through leakage detection, and reconstructs resolved events as forecasting instances. Discrete answer normalization and hierarchical scoring enable unified evaluation. We evaluate 27 model-configuration pairs on 300 FutureX-Past-derived questions over 24,300 prediction trials, assessing forecasting quality, sampling stability, and cost efficiency under controlled information boundaries. An audit finds only 1.37% residual leakage in retrieval results after temporal masking and leakage filtering. OracleProto retains resolved forecasting tasks as reusable evaluation resources under explicit temporal boundaries, supporting continual accumulation of forecasting datasets and systematic evaluation of forecasting capabilities. Code is available at this https URL.

Artificial Intelligence (cs.AI)

arXiv:2605.03762 [cs.AI]

(or arXiv:2605.03762v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2605.03762 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.