analiza aiqbench
The paper tests whether the gains in LLM-guided prompt and program optimization come from keeping a surviving parent candidate or from showing the proposer more failure cases per round. Across 1,750 pre-registered runs on nine benchmarks, greedy parent inheritance accounts for most of the progress, and broader feedback mainly speeds up early rounds before the advantage disappears. It also finds that proposer reflection is a small share of GEPA's token cost, while validation and minibatch gating consume most of the budget. For practitioners, this suggests that simpler inheritance-based loops may match more elaborate reflective pipelines at much lower compute, though the results are limited to the benchmarks and budgets studied.
Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization
arXiv:2610.41157v1 Announce Type: cross Abstract: Iterative language-model optimizers refine prompts and programs by evaluating a candidate, showing a proposer model a sample of failing cases, and keeping revisions that improve performance. Recent work attributes the limits of this loop to narrow single-round reflection, introducing full-set error clustering, targeted minibatches, or larger proposer models so the optimizer sees more failure modes per step. Yet every existing evaluation tests these mechanisms exclusively inside lineage-preserving search, where each revision edits a surviving parent, leaving the contribution of parent inheritance confounded with the breadth of per-round feedback. Across 1,750 pre-registered runs spanning nine prompt and program synthesis benchmarks, we separate parent inheritance from feedback disclosure under matched evaluation budgets. We find that greedy parent inheritance drives the vast majority of optimization progress, outperforming seed-reset resampling even when seed-reset proposals receive a clustered summary of all failures in every round or use a larger proposer model. While full-set error clustering and diverse failure selection improve single-step rewrites from the seed and accelerate the opening rounds of search, sequential parent inheritance rapidly closes this gap as solved error modes are retained in the candidate and filtered from subsequent failure lists. Moreover, decomposing token expenditure in GEPA reveals that proposer reflection accounts for only a small fraction of total cost whereas validation and minibatch gating consume nearly the entire budget, showing that greedy parent inheritance over raw failures captures the gains of complex reflective loops at a fraction of their compute.
Abstract:Iterative language-model optimizers refine prompts and programs by evaluating a candidate, showing a proposer model a sample of failing cases, and keeping revisions that improve performance. Recent work attributes the limits of this loop to narrow single-round reflection, introducing full-set error clustering, targeted minibatches, or larger proposer models so the optimizer sees more failure modes per step. Yet every existing evaluation tests these mechanisms exclusively inside lineage-preserving search, where each revision edits a surviving parent, leaving the contribution of parent inheritance confounded with the breadth of per-round feedback. Across 1,750 pre-registered runs spanning nine prompt and program synthesis benchmarks, we separate parent inheritance from feedback disclosure under matched evaluation budgets. We find that greedy parent inheritance drives the vast majority of optimization progress, outperforming seed-reset resampling even when seed-reset proposals receive a clustered summary of all failures in every round or use a larger proposer model. While full-set error clustering and diverse failure selection improve single-step rewrites from the seed and accelerate the opening rounds of search, sequential parent inheritance rapidly closes this gap as solved error modes are retained in the candidate and filtered from subsequent failure lists. Moreover, decomposing token expenditure in GEPA reveals that proposer reflection accounts for only a small fraction of total cost whereas validation and minibatch gating consume nearly the entire budget, showing that greedy parent inheritance over raw failures captures the gains of complex reflective loops at a fraction of their compute.
17 pages, 2 figures, 10 tables. Code, pre-registrations, and 1,750 run logs at this https URL
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.41157 [cs.SE]
(or arXiv:2610.41157v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2610.41157 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 356 cuvinte
Subiecte conexe
ISO: An RLVR-Native Optimization Stack
arXiv · 23 iul. 2026
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
arXiv · 15 sept. 2026
Optimizers for Diffusion Models: A Controlled Benchmark
arXiv · 22 sept. 2026
Learning to Control: The iUzawa-Net for Nonsmooth Optimal Control of Linear PDEs
arXiv · 28 oct. 2026