analiza aiqbench
This research introduces AgentRx, an automated diagnostic framework designed to localize failure steps in complex AI agent trajectories using constraint synthesis and LLM-based judges. By releasing a new benchmark of 170 annotated failures across diverse tasks, the study significantly improves step localization accuracy while establishing a cross-domain taxonomy for agent errors. This work directly addresses the critical need for reliability and interpretability in long-horizon, multi-agent systems.
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an $\textit{automated diagnostic framework}$ that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Abstract:AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an $\textit{automated diagnostic framework}$ that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Artificial Intelligence (cs.AI)
arXiv:2602.02475 [cs.AI]
(or arXiv:2602.02475v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2602.02475 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 240 cuvinte
Subiecte conexe
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
arXiv · 5 aug. 2026
DeFA: Dependency-Guided Failure Attribution for LLM Agents
arXiv · 2 oct. 2026
ARIA: Runtime Inspection and Amendment for Multi-Agent Systems
arXiv · 17 oct. 2026
When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation
arXiv · 24 aug. 2026