aiqbench analysis
Current benchmarks for Tool-Integrated Reasoning largely ignore real-world tool failures, creating a gap between idealized performance and actual agent reliability. New findings reveal that dynamic replanning is a distinct bottleneck where fault tolerance scales significantly slower than basic model size increases, particularly when agents encounter implicit semantic errors. This highlights the urgent need for robust error recovery mechanisms that go beyond simple scaling or prompting strategies.
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
arXiv:2606.05806v1 Announce Type: new Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.
Abstract:Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at this https URL.
Artificial Intelligence (cs.AI)
arXiv:2606.05806 [cs.AI]
(or arXiv:2606.05806v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.05806 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 232 words
Related coverage
[AINews] not much happened today
Latent Space · Oct 3, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
arXiv · Sep 29, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026