analiza aiqbench
This bibliometric audit reveals a widening "publication elicitation gap" where academic papers increasingly evaluate LLMs that are significantly behind current frontier capabilities, with the median lag growing by over 5 ECI points annually. The study highlights critical transparency issues, noting that explicit evaluation dates are rare and model-specific conclusions are often replaced by vague class-level claims, which undermines the reproducibility and relevance of AI research findings. To address this, the authors propose a distributed solution involving a new reporting checklist for authors, stricter enforcement by editors, and funder mandates for API access to verify model configurations.
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
arXiv:2605.04135v3 Announce Type: replace-cross Abstract: LLM evaluations in applied domains tend to reflect models that were already outclassed at time of publication. We observe a publication elicitation gap: the distance between the AI systems generating the results reported in an academic paper and the AI systems that a current reader of that paper would reasonably assume are being referenced. We systematically sweep OpenAlex from 2022-01-01 to 2026-04-01 (n = 112,303 LLM keyword matches). Then, we identify what models were evaluated (n = 18,574 admissible records). We then rank each evaluated LLM against a frontier LLM based on the Epoch AI Capabilities Index (ECI), an aggregate LLM capability score. At time of evaluation, the median paper is evaluating models that are behind frontier LLMs in capability, with a median gap of +10.85 ECI (H1; n = 12,312). This gap is growing, increasing at a rate of +5.53 ECI per year (H2, nominal 95% CI [+5.03, +5.83]). The sign holds even in the absence of any imputation for evaluation date. In papers (n = 728) where the date of evaluation is explicit and the model in question can be resolved to an ECI score, the median gap for H1 is +5.01 ECI. An explicitly stated evaluation date can be found in only 18.4% of full-text papers. After correction, in 52.5% (95% CI: [48.2, 56.9]) of abstracts in our audit, conclusions are stated at the class level ("AI") rather than the model level. For papers about reasoning models, only 3.2% of abstracts and 21.2% of full-text articles disclose the reasoning mode status of the models used (H4). We propose a solution to this problem that is distributed among authors, editors, and funders. First, reporting from authors. VERSIO-AI v1.2 is a proposed 13-item checklist to cover the configuration surface described herein. Second, enforcement from journal editors and peer reviewers. Third, conditioning grants on disclosure and providing API access.
Abstract:LLM evaluations in applied domains tend to reflect models that were already outclassed at time of publication. We observe a publication elicitation gap: the distance between the AI systems generating the results reported in an academic paper and the AI systems that a current reader of that paper would reasonably assume are being referenced. We systematically sweep OpenAlex from 2022-01-01 to 2026-04-01 (n = 112,303 LLM keyword matches). Then, we identify what models were evaluated (n = 18,574 admissible records). We then rank each evaluated LLM against a frontier LLM based on the Epoch AI Capabilities Index (ECI), an aggregate LLM capability score. At time of evaluation, the median paper is evaluating models that are behind frontier LLMs in capability, with a median gap of +10.85 ECI (H1; n = 12,312). This gap is growing, increasing at a rate of +5.53 ECI per year (H2, nominal 95% CI [+5.03, +5.83]). The sign holds even in the absence of any imputation for evaluation date. In papers (n = 728) where the date of evaluation is explicit and the model in question can be resolved to an ECI score, the median gap for H1 is +5.01 ECI. An explicitly stated evaluation date can be found in only 18.4% of full-text papers. After correction, in 52.5% (95% CI: [48.2, 56.9]) of abstracts in our audit, conclusions are stated at the class level ("AI") rather than the model level. For papers about reasoning models, only 3.2% of abstracts and 21.2% of full-text articles disclose the reasoning mode status of the models used (H4). We propose a solution to this problem that is distributed among authors, editors, and funders. First, reporting from authors. VERSIO-AI v1.2 is a proposed 13-item checklist to cover the configuration surface described herein. Second, enforcement from journal editors and peer reviewers. Third, conditioning grants on disclosure and providing API access.
63 pages, 9 figures, 9 tables. v3: corrects the validation-sample, primary-model and appendix-reference errors; revised text; updated declarations. Pre-registered on OSF: this https URL. Code: this https URL. VERSIO-AI v1.2 reporting checklist: this https URL. frontierlag package + per-DOI audit tool: this https URL
Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.04135 [cs.CY]
(or arXiv:2605.04135v3 [cs.CY] for this version)
Text integral sub licența deschisă a publicației · 372 cuvinte
Subiecte conexe
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
arXiv · 6 iun. 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026
[AINews] not much happened today
Latent Space · 10 sept. 2026
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026