aiqbench analysis
The paper introduces a two-dimensional framework for testing how LLM agents handle noisy or corrupted inputs, and reports that perturbations consistently degrade performance on APEX-Agents and Toolathlon. Its most useful finding for practitioners is that agents rarely notice altered inputs and often pass corrupted information downstream without checking it, while successful perturbed runs can cost substantially more compute. These are results on two benchmarks, so the vulnerability patterns should be verified on other task environments before being generalized.
AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations
arXiv:2610.21624v1 Announce Type: new Abstract: Real-world tasks are often complex and noisy, posing substantial challenges for LLM agents. Yet existing evaluations lack a systematic framework for jointly characterizing potential perturbation targets and types. We use this two-dimensional view to conduct a unified study that quantifies the robustness of current frontier models, traces the mechanisms behind perturbation-induced failures, and draws implications for robustness-oriented training. Through controlled evaluations on APEX-Agents and Toolathlon, we find that perturbations consistently degrade performance, while the dominant vulnerabilities vary across task environments. Trajectory-level analysis reveals that agents rarely recognize altered inputs and often propagate corrupted information without verification. A focused cost analysis further shows that successful perturbed trajectories can require substantially more computation. These findings motivate training objectives that cover diverse forms of noise and reward verification, uncertainty awareness, and recovery. We provide a rigorous benchmark-construction pipeline. Code and data will be released at https://github.com/xuyj233/AgentPerturb.
Abstract:Real-world tasks are often complex and noisy, posing substantial challenges for LLM agents. Yet existing evaluations lack a systematic framework for jointly characterizing potential perturbation targets and types. We use this two-dimensional view to conduct a unified study that quantifies the robustness of current frontier models, traces the mechanisms behind perturbation-induced failures, and draws implications for robustness-oriented training. Through controlled evaluations on APEX-Agents and Toolathlon, we find that perturbations consistently degrade performance, while the dominant vulnerabilities vary across task environments. Trajectory-level analysis reveals that agents rarely recognize altered inputs and often propagate corrupted information without verification. A focused cost analysis further shows that successful perturbed trajectories can require substantially more computation. These findings motivate training objectives that cover diverse forms of noise and reward verification, uncertainty awareness, and recovery. We provide a rigorous benchmark-construction pipeline. Code and data will be released at this https URL.
25 pages, 10 figures. Project repository: this https URL
Artificial Intelligence (cs.AI)
arXiv:2610.21624 [cs.AI]
(or arXiv:2610.21624v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.21624 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 248 words
Related coverage
Alignment Midtraining Cracks Under Pressure
LessWrong · Sep 21, 2026
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · Sep 24, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026