aiqbench analysis
This audit exposes a critical flaw in AI agent evaluation pipelines where reported metrics often measure system artifacts rather than actual performance, highlighting the need for rigorous verification of tool-attempt data and timing statistics. By reproducing calculations and analyzing missing values, the study provides a reusable framework to disentangle gate policies from genuine request-level accounting in complex agent systems.
When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
arXiv:2609.12017v1 Announce Type: cross Abstract: Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.
Abstract:Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.
22 pages, 4 figures. Retrospective measurement audit with aggregate data, offline analysis code, and a prospective, unexecuted operator-coverage catalog in ancillary files
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2609.12017 [cs.SE]
(or arXiv:2609.12017v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2609.12017 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 268 words
Related coverage
Why do models task game?
AI Alignment Forum · Aug 6, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026
Article: The Platform Engineering Playbook for Production LLMs
InfoQ · Oct 5, 2026