aiqbench analysis
This study critically examines the long-term accessibility of GitHub repositories linked in computational linguistics papers, revealing that recent publications face similar availability issues as older ones due to a rise in empty or placeholder repos. The findings highlight a significant reproducibility crisis within NLP research where code sharing practices are not improving over time, posing challenges for verification and reuse in machine learning workflows.
Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications
arXiv:2609.34534v1 Announce Type: new Abstract: Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.
Abstract:Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.
Accepted for publication in Computational Linguistics. Author's final version (pre-MIT Press publication)
Computation and Language (cs.CL)
arXiv:2609.34534 [cs.CL]
(or arXiv:2609.34534v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.34534 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 236 words
