AI Evaluation Should Require Standardized Item-Level Data Releases

arXiv:2604.03244v3 Announce Type: replace Abstract: This position paper argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization; a disproportionate focus on aggregate model scores compounds these failures by keeping them undetected and unfixable. Without item-level evidence, validity claims cannot be assessed, resulting in inflated capability claims, misdirected research, and unwarranted trust in deployed systems. Our position is that valid evaluation requires empirical evidence from item-level benchmark data, including item content, test conditions, model responses, and item-level score statistics, and that the standardized release of such data should be treated as core AI evaluation infrastructure. Such releases also enable transparency, replicability, and auditability of evaluation results. To show that the norm is both feasible and consequential, we construct OpenEval, an item-level repository of 10.7M+ responses across 160K+ items from widely-used benchmarks, under a unified schema that the AI evaluation community can build upon. We demonstrate how item-level data can identify low-quality items, document construct misalignment, and recover validity evidence about benchmarks' internal structure. We address objections concerning contamination and coordination burden, and show that both are tractable relative to the costs of decisions made on inadequately validated evaluative claims.

Abstract:This position paper argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization; a disproportionate focus on aggregate model scores compounds these failures by keeping them undetected and unfixable. Without item-level evidence, validity claims cannot be assessed, resulting in inflated capability claims, misdirected research, and unwarranted trust in deployed systems. Our position is that valid evaluation requires empirical evidence from item-level benchmark data, including item content, test conditions, model responses, and item-level score statistics, and that the standardized release of such data should be treated as core AI evaluation infrastructure. Such releases also enable transparency, replicability, and auditability of evaluation results. To show that the norm is both feasible and consequential, we construct OpenEval, an item-level repository of 10.7M+ responses across 160K+ items from widely-used benchmarks, under a unified schema that the AI evaluation community can build upon. We demonstrate how item-level data can identify low-quality items, document construct misalignment, and recover validity evidence about benchmarks' internal structure. We address objections concerning contamination and coordination burden, and show that both are tractable relative to the costs of decisions made on inadequately validated evaluative claims.

Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Databases (cs.DB)

arXiv:2604.03244 [cs.AI]

(or arXiv:2604.03244v3 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2604.03244 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

AI Evaluation Should Require Standardized Item-Level Data Releases

Related coverage