Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models

arXiv:2605.24564v2 Announce Type: replace Abstract: Backtesting large language models (LLMs) on historical financial data is unreliable when their pre-training data include the evaluated events. An LLM trained in 2024 may already encode how stocks moved during 2018-2020. We name this failure parametric look-ahead bias and propose FinCAD, an inference-time adaptation of Context-Aware Decoding that attenuates contributions from memorised historical outcomes without retraining. FinCAD pairs an adversarial bias-discovery pipeline that learns a model-specific memory-activating prior prompt with an entity- and date-adaptive rule that scales the CAD strength using a per-(entity, date) confidence signal. Across five 7-14B LLMs and five mega-cap equities, the largest model-level mean in-sample return correction is -67.1%. For the three larger models, 2025 out-of-sample returns remain within $8K and mean Sharpe within $\pm$0.10 of baseline; mean general-benchmark accuracy remains positive or within -1.7 points for four of five models. On an eleven-model leaderboard, FinCAD raises the subset-averaged in-sample/out-of-sample Spearman correlation from +0.779 to +0.846, yielding rankings that are more closely aligned with post-cutoff performance.

Abstract:Backtesting large language models (LLMs) on historical financial data is unreliable when their pre-training data include the evaluated events. An LLM trained in 2024 may already encode how stocks moved during 2018-2020. We name this failure parametric look-ahead bias and propose FinCAD, an inference-time adaptation of Context-Aware Decoding that attenuates contributions from memorised historical outcomes without retraining. FinCAD pairs an adversarial bias-discovery pipeline that learns a model-specific memory-activating prior prompt with an entity- and date-adaptive rule that scales the CAD strength using a per-(entity, date) confidence signal. Across five 7-14B LLMs and five mega-cap equities, the largest model-level mean in-sample return correction is -67.1%. For the three larger models, 2025 out-of-sample returns remain within $8K and mean Sharpe within $\pm$0.10 of baseline; mean general-benchmark accuracy remains positive or within -1.7 points for four of five models. On an eleven-model leaderboard, FinCAD raises the subset-averaged in-sample/out-of-sample Spearman correlation from +0.779 to +0.846, yielding rankings that are more closely aligned with post-cutoff performance.

Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Machine Learning (cs.LG)

arXiv:2605.24564 [cs.AI]

(or arXiv:2605.24564v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2605.24564 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models

Related coverage