analiza aiqbench
This study critically evaluates the predictive validity of commonsense benchmarks in Large Language Models, finding they offer only task-dependent rather than broad evidence of real-world competence. The research concludes that revised benchmark variants fail to significantly improve downstream prediction power compared to established standards. For researchers relying on these metrics for model selection or capability assessment, this highlights a persistent gap between standardized test scores and actual pragmatic reasoning abilities in LLMs.
Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks
arXiv:2608.03340v1 Announce Type: new Abstract: Predicting LLM's capabilities on real-world tasks is essential, yet the extent to which performance on commonsense benchmarks predicts downstream performance remains underspecified. To establish the practical usability of widely adopted commonsense benchmarks, we evaluate 23 models from six families on four established commonsense benchmarks, four reworked variants, three non-commonsense controls, and eight downstream tasks requiring implicit social, pragmatic, temporal, or physical reasoning. We compare model rankings, compute controlled correlations, and use leave-one-family-out cross-validation to assess the criterion validity of commonsense benchmarks. Our results show that revised benchmarks largely preserve original model rankings and do not improve downstream predictive power. Commonsense benchmarks show consistent cross-family predictive validity for only a narrow subset of downstream tasks, with smaller or metric-specific gains elsewhere. Overall, standardized commonsense benchmarks provide task-dependent rather than broad evidence of downstream commonsense competence.
Abstract:Predicting LLM's capabilities on real-world tasks is essential, yet the extent to which performance on commonsense benchmarks predicts downstream performance remains underspecified. To establish the practical usability of widely adopted commonsense benchmarks, we evaluate 23 models from six families on four established commonsense benchmarks, four reworked variants, three non-commonsense controls, and eight downstream tasks requiring implicit social, pragmatic, temporal, or physical reasoning. We compare model rankings, compute controlled correlations, and use leave-one-family-out cross-validation to assess the criterion validity of commonsense benchmarks. Our results show that revised benchmarks largely preserve original model rankings and do not improve downstream predictive power. Commonsense benchmarks show consistent cross-family predictive validity for only a narrow subset of downstream tasks, with smaller or metric-specific gains elsewhere. Overall, standardized commonsense benchmarks provide task-dependent rather than broad evidence of downstream commonsense competence.
Computation and Language (cs.CL)
arXiv:2608.03340 [cs.CL]
(or arXiv:2608.03340v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.03340 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 229 cuvinte
Subiecte conexe
ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
arXiv · 17 iun. 2026
Manac\'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation
arXiv · 1 sept. 2026
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · 8 oct. 2026
[AINews] not much happened today
Latent Space · 18 sept. 2026