Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

arXiv:2608.00794v1 Announce Type: new Abstract: Agentic AI systems are evaluated using automated benchmarks whose scores justify deployment decisions, safety certifications, and regulatory compliance claims. We present an empirical analysis demonstrating that these scores are systematically less trustworthy than current practice acknowledges. The problem operates at three compounding layers. First, tasks are increasingly generated by language models: audits of ten popular benchmarks found validity flaws in seven and reporting gaps in all ten. Second, human users are replaced by LLM simulators, but calibration studies document inter-simulator variance up to 9 percentage points and systematic directional miscalibration, particularly for non-Standard American English speakers. Third, our structured survey of 55 papers finds that approximately 82% apply structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics. These failures compound multiplicatively rather than additively. Under independence, a pipeline retaining 70% of valid signal at task generation, 80% at simulation, and 65% at judgment is at most 36% valid against the intended construct; the bound spans 0.22--0.54 across the empirical estimate range. We formalize this as $V_{\text{total}} \leq V_1 \times V_2 \times V_3$ and show it tightens further under correlated failures when the same model family operates across all three layers. We derive eight prescriptions grounded in psychometric science: a simulation calibration floor of $\text{ICC}(A,1) \geq 0.70$; domain-stratified reliability thresholds ($\alpha \geq 0.67$ / $0.70$ / $0.80$ by consequence level); structured IRR metric selection rules based on pipeline design; and IRR as a mandatory reporting field. The measurement tools exist; the field's task is to apply them.

Abstract:Agentic AI systems are evaluated using automated benchmarks whose scores justify deployment decisions, safety certifications, and regulatory compliance claims. We present an empirical analysis demonstrating that these scores are systematically less trustworthy than current practice acknowledges. The problem operates at three compounding layers. First, tasks are increasingly generated by language models: audits of ten popular benchmarks found validity flaws in seven and reporting gaps in all ten. Second, human users are replaced by LLM simulators, but calibration studies document inter-simulator variance up to 9 percentage points and systematic directional miscalibration, particularly for non-Standard American English speakers. Third, our structured survey of 55 papers finds that approximately 82% apply structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics. These failures compound multiplicatively rather than additively. Under independence, a pipeline retaining 70% of valid signal at task generation, 80% at simulation, and 65% at judgment is at most 36% valid against the intended construct; the bound spans 0.22--0.54 across the empirical estimate range. We formalize this as $V_{\text{total}} \leq V_1 \times V_2 \times V_3$ and show it tightens further under correlated failures when the same model family operates across all three layers. We derive eight prescriptions grounded in psychometric science: a simulation calibration floor of $\text{ICC}(A,1) \geq 0.70$; domain-stratified reliability thresholds ($\alpha \geq 0.67$ / $0.70$ / $0.80$ by consequence level); structured IRR metric selection rules based on pipeline design; and IRR as a mandatory reporting field. The measurement tools exist; the field's task is to apply them.

22 pages, 2 figures, 4 tables. Submitted to a peer-reviewed journal

Artificial Intelligence (cs.AI)

arXiv:2608.00794 [cs.AI]

(or arXiv:2608.00794v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2608.00794 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

Subiecte conexe