How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation

arXiv:2606.00920v2 Announce Type: replace-cross Abstract: A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using 16 models on 99 recent LeetCode-style problems with two prompts and five runs each (15,840 judged runs in a fully crossed design), we estimate both components and ask a design question: how much can repeated running buy? First, the pooled-binomial interval that repeated-run protocols invite answers neither question of interest---it is about $2\times$ too narrow for generalisation to new problems and $2\times$ too wide for the fixed problem set---and under correctly clustered intervals, 12 of 15 adjacent leaderboard pairs are statistically unresolved. Second, the variance ceiling $1+\mu_w/\sigma_b^2$, the largest factor by which repetition can reduce score variance, has median 1.34 across models (range 1.05--1.71), so repetition can narrow confidence intervals by at most 23% on this benchmark, and by 14% for the median model. Third, once the cost of adding verified problems is priced in, the precision-optimal repeat count is $R^{*}=\sqrt{\mu_w c_p/(\sigma_b^2 c_g)}$, and the variance curve is flat around it: on this benchmark the decisive reason to repeat is not precision but identification---two to three runs suffice to estimate the components that the ceiling and the fixed-benchmark interval require. All three are measurements of one benchmark at one decoding temperature, and we state explicitly what they do and do not license.

Abstract:A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using 16 models on 99 recent LeetCode-style problems with two prompts and five runs each (15,840 judged runs in a fully crossed design), we estimate both components and ask a design question: how much can repeated running buy? First, the pooled-binomial interval that repeated-run protocols invite answers neither question of interest---it is about $2\times$ too narrow for generalisation to new problems and $2\times$ too wide for the fixed problem set---and under correctly clustered intervals, 12 of 15 adjacent leaderboard pairs are statistically unresolved. Second, the variance ceiling $1+\mu_w/\sigma_b^2$, the largest factor by which repetition can reduce score variance, has median 1.34 across models (range 1.05--1.71), so repetition can narrow confidence intervals by at most 23% on this benchmark, and by 14% for the median model. Third, once the cost of adding verified problems is priced in, the precision-optimal repeat count is $R^{*}=\sqrt{\mu_w c_p/(\sigma_b^2 c_g)}$, and the variance curve is flat around it: on this benchmark the decisive reason to repeat is not precision but identification---two to three runs suffice to estimate the components that the ceiling and the fixed-benchmark interval require. All three are measurements of one benchmark at one decoding temperature, and we state explicitly what they do and do not license.

17 pages, 4 figures, 11 tables. Substantially revised and retitled from v1; reframes the original repeated-run evaluation data as a variance decomposition, with expanded statistical validation and experimental details. Data and code: this https URL . Accepted at the NeurIPS 2026 Workshop on AI for Verifiable Coding (VeriCodeGen)

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)

arXiv:2606.00920 [cs.LG]

(or arXiv:2606.00920v2 [cs.LG] for this version)

analiza aiqbench

How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation

Subiecte conexe