analiza aiqbench
This large-scale study of 56,800 papers from top AI conferences reveals a significant shift toward open science over the last decade, driven more by community momentum than formal checklists. Documentation practices have improved markedly, with code and data sharing increasing sixfold between 2014 and 2024, suggesting that reproducibility rates may be rising alongside these behavioral changes.
The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers
arXiv:2606.16974v1 Announce Type: new Abstract: The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.
Abstract:The reproducibility crisis has directed the AI research community toward improving documentation practices. Several studies have identified methodological issues, and in response, the most impactful venues in the field have introduced reproducibility checklists. We seek to understand whether documentation practices have changed over time by assessing all published papers at five leading AI conferences over the past decade. Seven reproducibility variables were identified, quality-assured and used to analyse 56 800 publications. Our analysis reveals that in the period 2014 to 2024, documentation practices have improved; papers sharing both code and data increased nearly sixfold, from 11% to 64% Building on empirical reproducibility rates from a prior study, we estimate - inferred from documentation practices, not direct testing - that reproducibility increased from 28% in 2014 to 64% in 2024. Improvements in documentation practices predate the introduction of reproducibility checklists, suggesting these changes reflect a broader movement toward open science rather than a direct response to formal requirements.
Artificial Intelligence (cs.AI)
arXiv:2606.16974 [cs.AI]
(or arXiv:2606.16974v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.16974 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 251 cuvinte
Subiecte conexe
Beyond Reproducibility: Towards Security-Aware Evaluation of Research Artifacts
arXiv · 5 sept. 2026
Unsupervised Clustering for Fault Analysis in High-Voltage Power Systems Using Voltage and Current Signals
arXiv · 15 sept. 2026
ST-Bench: A Spatial-Temporal Benchmark for Multi-Agent System Generation on Scientific Research Tasks
arXiv · 7 oct. 2026
Comparing and Modeling Argumentation in German Political Communication across Arenas
arXiv · 4 aug. 2026