analiza aiqbench
This arXiv paper is about AI: it studies auto-harness systems that improve LLM agents by adapting prompts, skills, and tools over continuous task streams. The reported gains (for example, 22.2% to 80.9% market accuracy with the same solver) come from a self-reported benchmark on recorded data, so they are best read as evidence for the approach rather than a general result. The more useful finding for practitioners is that accumulated artifacts don't guarantee sustained improvement, and that continued revision of stored knowledge may be needed to avoid losing earlier gains.
Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
arXiv:2606.01770v3 Announce Type: replace Abstract: Auto-harness systems improve LLM agents by adapting prompts, skills, tools, and supporting infrastructure from execution feedback. Sustaining these gains on open-ended task streams requires handling growing histories, heterogeneous tasks, and shifting distributions. We introduce Adaptive Auto-Harness, a unified framework that separates persistent evolution workspaces from deployable harnesses and supports interchangeable evolution procedures. We compare configurations adapted from five prior approaches on recorded streams of prediction markets, security challenges, and event forecasts. With the same Sonnet solver, our full system reaches 80.9\% market all-task accuracy and 50.2\% security Pass@1, versus 22.2\% and 37.2\% without evolution; market accuracy also exceeds a market-following control's 72.0\%. Our analyses show that artifact accumulation does not guarantee sustained gains, and knowledge that transfers across tasks can still lose value as the task distribution changes. Continued evolution can therefore help preserve earlier gains through revision of accumulated knowledge. Code: https://github.com/A-EVO-Lab/a-evolve.
Abstract:Auto-harness systems improve LLM agents by adapting prompts, skills, tools, and supporting infrastructure from execution feedback. Sustaining these gains on open-ended task streams requires handling growing histories, heterogeneous tasks, and shifting distributions. We introduce Adaptive Auto-Harness, a unified framework that separates persistent evolution workspaces from deployable harnesses and supports interchangeable evolution procedures. We compare configurations adapted from five prior approaches on recorded streams of prediction markets, security challenges, and event forecasts. With the same Sonnet solver, our full system reaches 80.9\% market all-task accuracy and 50.2\% security Pass@1, versus 22.2\% and 37.2\% without evolution; market accuracy also exceeds a market-following control's 72.0\%. Our analyses show that artifact accumulation does not guarantee sustained gains, and knowledge that transfers across tasks can still lose value as the task distribution changes. Continued evolution can therefore help preserve earlier gains through revision of accumulated knowledge. Code: this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.01770 [cs.LG]
(or arXiv:2606.01770v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.01770 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 240 cuvinte