aiqbench analysis
HERA addresses the critical reliability gap in LLM agents by introducing a framework for harness-environment co-evolution to improve agentic abstention. By automatically generating verifiable task pairs through environment mutations, the method significantly boosts accuracy in recognizing when no valid solution exists while maintaining high completion rates for feasible tasks. This approach demonstrates strong generalization across 19 different models and enables smaller agents to match larger ones at reduced cost, offering a robust path forward for safe tool-use environments.
HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
arXiv:2610.06563v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.
Abstract:Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.
23 pages. Project page: this https URL
Artificial Intelligence (cs.AI)
arXiv:2610.06563 [cs.AI]
(or arXiv:2610.06563v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.06563 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 311 words
Related coverage
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
WEFT: Scaling Tool-Use Post-Training for General-Purpose Agents
arXiv · Sep 30, 2026
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
arXiv · Jun 17, 2026
Bayesian-Agent: Posterior-Guided Skill Evolution for LLM Agent Harnesses
arXiv · Jun 9, 2026