aiqbench analysis
This study quantifies the inherent noise floor in agent benchmarks by comparing deterministic reruns against semantic prompt perturbations, revealing that measurement variability is driven significantly more by input instability than model stochasticity. The findings highlight a critical gap where marginal accuracy metrics obscure distinct failure modes, such as malformed outputs, which vary drastically across different provider endpoints. For researchers evaluating autonomous agents, this audit underscores the necessity of distinguishing between true capability limits and artifacts introduced by benchmarking methodology.
Noise Floor Audit for Agent Benchmarks
arXiv:2608.22331v1 Announce Type: new Abstract: We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq endpoints and a thinking-enabled Gemini setting: ever-flip fractions are 0.7%, 2.0%, and 2.7%, with mean run correlations of 0.997, 0.966, and 0.961. Semantics-preserving prompt perturbations create the larger floor on all endpoints, with median perturbation paired SDs 11x to 58x larger than rerun paired SDs. The failure character also shifts: malformed-output failures account for 30%, 7%, and <1% of task failures, so marginal accuracy hides not only stability but also failure mode.
Abstract:We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq endpoints and a thinking-enabled Gemini setting: ever-flip fractions are 0.7%, 2.0%, and 2.7%, with mean run correlations of 0.997, 0.966, and 0.961. Semantics-preserving prompt perturbations create the larger floor on all endpoints, with median perturbation paired SDs 11x to 58x larger than rerun paired SDs. The failure character also shifts: malformed-output failures account for 30%, 7%, and <1% of task failures, so marginal accuracy hides not only stability but also failure mode.
Computation and Language (cs.CL)
arXiv:2608.22331 [cs.CL]
(or arXiv:2608.22331v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.22331 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 200 words
Related coverage
Agentic AI for Gravitational Wave Data Analysis: A Head-to-Head Comparison of Coding Agents Executing a Matched Filter Pipeline on Einstein Telescope Simulated Data
arXiv · Sep 16, 2026
CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents
arXiv · Aug 5, 2026
News Archive 2026
Quantum Computing Report · Jul 25, 2026
How the Audit Rule Shapes Faithful Factor Explanations in LLMs
arXiv · Oct 2, 2026