aiqbench analysis
This paper introduces a rigorous paired benchmark to evaluate Large Language Models in auditing financial backtests, specifically addressing the critical trade-off between flaw recall and false positives on clean controls. By isolating single methodology changes while holding strategy and code style constant, the study demonstrates that current models often over-flag valid strategies unless equipped with specific "clean-aware" calibration mechanisms. The findings highlight that reporting clean-control rates is essential for distinguishing model performance, as recall metrics alone fail to capture significant differences in reliability across different endpoints.
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
arXiv:2609.28090v1 Announce Type: cross Abstract: Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
Abstract:Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Software Engineering (cs.SE)
arXiv:2609.28090 [cs.CL]
(or arXiv:2609.28090v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.28090 arXiv-issued DOI via DataCite (pending registration)
From: Vladislav Smirnov Mr. [view email] [v1] Wed, 23 Sep 2026 13:25:25 UTC (187 KB)
Full text under the publisher's open licence · 201 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
If This Were a Test, How Much Would It Cost?
LessWrong · Jun 16, 2026
Academia is for Ambition — Alex Zhang, MIT
Latent Space · Oct 2, 2026