aiqbench analysis
This research demonstrates that using a deterministic, judge-free build check as a filter significantly outperforms learned reward models in code generation self-distillation. By prioritizing verifier precision over lenient checks, the method achieves superior generalization on held-out tasks without optimizing for proxy features that do not improve actual artifacts. The findings highlight that search efficiency and model improvement are directly governed by the strictness of the verification gate rather than candidate volume.
The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillation
arXiv:2607.09709v2 Announce Type: replace Abstract: Post-training a code generator against a learned judge can optimize proxy features that raise the score without improving the artifact. We study the opposite signal: a deterministic, judge-free filter that asks only whether a generated project launches cleanly under a headless engine (strict-launch). Under this gate, rejection-sampling self-distillation compounds out-of-family generalization: on GameCraft-Bench a 14B model raises the per-candidate clean-launch rate on four held-out families from 8.8% to 42.2% and coverage at 32 candidates from 84% to 100%, the gold references' own ceiling, beating the supervised model on every one of the 25 held-out tasks. The gate costs one engine invocation per candidate: no reward model, no judge. At a fixed admitted count, what governs the loop is verifier precision. Swapping in a lenient build check alone erases the gain (p=0.0012); a matched gold-duplication control regresses below the supervised model. Under a semantic gate on APPS, dialing fuel precision from 1.0 to 0.25 at fixed candidate count prices that fuel linearly: over 23 training seeds, half-clean fuel returns +3.59 percentage points against the +3.69 a linear rate predicts. Under count-matched rejection-SFT only one direction of verifier error carries a measurable cost. Search obeys the same gate: quadrupling the harvest budget is worth +1.62pp behind a strict gate and nothing distinguishable from zero behind a partial-credit one. Recall is nearly free; search pays only through a precise gate: the verifier is the curriculum.
Abstract:Post-training a code generator against a learned judge can optimize proxy features that raise the score without improving the artifact. We study the opposite signal: a deterministic, judge-free filter that asks only whether a generated project launches cleanly under a headless engine (strict-launch). Under this gate, rejection-sampling self-distillation compounds out-of-family generalization: on GameCraft-Bench a 14B model raises the per-candidate clean-launch rate on four held-out families from 8.8% to 42.2% and coverage at 32 candidates from 84% to 100%, the gold references' own ceiling, beating the supervised model on every one of the 25 held-out tasks. The gate costs one engine invocation per candidate: no reward model, no judge. At a fixed admitted count, what governs the loop is verifier precision. Swapping in a lenient build check alone erases the gain (p=0.0012); a matched gold-duplication control regresses below the supervised model. Under a semantic gate on APPS, dialing fuel precision from 1.0 to 0.25 at fixed candidate count prices that fuel linearly: over 23 training seeds, half-clean fuel returns +3.59 percentage points against the +3.69 a linear rate predicts. Under count-matched rejection-SFT only one direction of verifier error carries a measurable cost. Search obeys the same gate: quadrupling the harvest budget is worth +1.62pp behind a strict gate and nothing distinguishable from zero behind a partial-credit one. Recall is nearly free; search pays only through a precise gate: the verifier is the curriculum.
15 pages, 8 figures, 6 tables. v2: substantially revised and extended (new title, new APPS experiments on verifier precision, unbiased coverage estimator, three training seeds)
Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
arXiv:2607.09709 [cs.AI]
(or arXiv:2607.09709v2 [cs.AI] for this version)
Full text under the publisher's open licence · 271 words
Related coverage
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
METR · Aug 26, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026