The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules

Abstract:Self-improving LLM systems propose changes to themselves and keep those that score better on a small evaluation set. We treat this keep-if-better step as selection under measurement noise, model the correlated errors of the candidates in a single decision, and study empirically what happens when the evaluation set is reused. In runs where Qwen models rewrite their own instructions and every candidate is also scored on 600 held-out items, most proposals after the first are harmful, and the model gives the size of the winner's curse of a generation's best candidate. With a prior from a separate pilot, it matches the average overstatement of first-generation commits in native loops, though not setting by setting. In a pre-registered study, the final selection-set score of greedy loops exceeded held-out accuracy by 13 to 20 points with 16 selection items and by 1 to 5 points with 256. Held-out gains grew with the selection set on TREC but not on GSM8K, and the tested acceptance rules did not beat greedy acceptance over whole runs. Gains measured on the selection set also exceeded held-out gains when a current model refined a competent instruction, and in the validation scores of GEPA and MIPROv2. Scoring the starting and the current instruction on 64 items never used for selection removes the average bias of a loop's reported gain, but single estimates remain off by about 6 points. Self-improvement studies should report held-out gains with their uncertainty.

Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2610.09239 [cs.AI]

(or arXiv:2610.09239v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.09239 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

winners_curse_supplement/README.md

winners_curse_supplement/code/accept.py

winners_curse_supplement/code/analyze_e1.py

winners_curse_supplement/code/analyze_gepa.py

winners_curse_supplement/code/analyze_mipro.py

winners_curse_supplement/code/analyze_rescore.py

winners_curse_supplement/code/analyze_rgepa.py

winners_curse_supplement/code/analyze_runs.py

winners_curse_supplement/code/analyze_strong.py

winners_curse_supplement/code/analyze_strong_start.py

winners_curse_supplement/code/analyze_v2.py

winners_curse_supplement/code/audit_gepa_logs.py

winners_curse_supplement/code/audit_report.py

winners_curse_supplement/code/check_pilot_priors.py

winners_curse_supplement/code/commit_prediction.py

winners_curse_supplement/code/decision_analysis_v2.py

winners_curse_supplement/code/duplicate_sensitivity.py

winners_curse_supplement/code/eval_reference.py

winners_curse_supplement/code/fig_breeder.py

winners_curse_supplement/code/fig_main.py

winners_curse_supplement/code/fig_systems.py

winners_curse_supplement/code/fig_v2.py

winners_curse_supplement/code/gen1_share.py

winners_curse_supplement/code/gepa_budget.py

winners_curse_supplement/code/gepa_predict.py

winners_curse_supplement/code/gepa_predict_val.py

winners_curse_supplement/code/gepa_study.py

winners_curse_supplement/code/gepa_truncate.py

winners_curse_supplement/code/llm.py

winners_curse_supplement/code/lockin_analysis.py

winners_curse_supplement/code/make_appendix_tables.py

winners_curse_supplement/code/make_decision_table_v2.py

winners_curse_supplement/code/make_gepa_follow_table.py

winners_curse_supplement/code/make_online_table.py

winners_curse_supplement/code/make_transfer_table.py

winners_curse_supplement/code/make_v2_table.py

winners_curse_supplement/code/mipro_study.py

winners_curse_supplement/code/online_causal.py

winners_curse_supplement/code/pilot_priors.py

winners_curse_supplement/code/power_v2.py

winners_curse_supplement/code/prep_data.py

winners_curse_supplement/code/proposal_stats.py

winners_curse_supplement/code/recon_plan.py

winners_curse_supplement/code/rescore_gsm8k.py

winners_curse_supplement/code/selfimprove.py

winners_curse_supplement/code/shortfall.py

winners_curse_supplement/code/tasks.py

winners_curse_supplement/code/theory.py

winners_curse_supplement/code/transfer_analysis.py

winners_curse_supplement/code/verify_gepa_logs.py

winners_curse_supplement/paper/figs/breeder.pdf

winners_curse_supplement/paper/figs/main.pdf

winners_curse_supplement/paper/figs/systems.pdf

winners_curse_supplement/paper/figs/v2.pdf

winners_curse_supplement/reproduce.py

winners_curse_supplement/requirements-experiments.txt

winners_curse_supplement/requirements.txt

winners_curse_supplement/scientific_data.tar.xz

winners_curse_supplement/scientific_data_manifest.json

winners_curse_supplement/tests/test_breeder_intervals.py

winners_curse_supplement/tests/test_e1_edge_cases.py

winners_curse_supplement/tests/test_gepa_rng.py

winners_curse_supplement/tests/test_gold_noise.py

winners_curse_supplement/tests/test_online_weighting.py

winners_curse_supplement/tests/test_theory.py

winners_curse_supplement/tests/test_unpack_data.py

winners_curse_supplement/unpack_data.py

(62 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.