analiza aiqbench
This paper reframes machine learning benchmarking not merely as a performance metric but as an epistemic practice requiring explicit validity assumptions to draw scientific inferences about research progress. By adapting psychological validity theory, the authors provide a framework for interpreting results from major datasets like ImageNet with greater rigor and clarity regarding the limits of predictability.
The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models
arXiv:2510.23191v2 Announce Type: replace Abstract: Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing substantial scientific inferences requires additional assumptions. Adapting ideas from psychological validity theory, we propose validity conditions that make these assumptions explicit. In two case studies---ImageNet and the Fragile Families Challenge---we show how benchmark results can support inferences about research progress and limits of predictability, situating predictive benchmarking as a distinct epistemic practice in machine learning.
Abstract:Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing substantial scientific inferences requires additional assumptions. Adapting ideas from psychological validity theory, we propose validity conditions that make these assumptions explicit. In two case studies---ImageNet and the Fragile Families Challenge---we show how benchmark results can support inferences about research progress and limits of predictability, situating predictive benchmarking as a distinct epistemic practice in machine learning.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2510.23191 [cs.LG]
(or arXiv:2510.23191v2 [cs.LG] for this version)
Freiesleben T, Zezulka S. The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models. Philosophy of Science. Published online 2026:1-30
https://doi.org/10.1017/psa.2026.10280 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 133 cuvinte
