Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

Abstract:Neuroscience data are fragmented across labs, experimental paradigms, and storage formats, and data reuse often requires substantial manual effort. Standardized formats have been proposed in response, but a fundamental tension persists: formats flexible enough for diverse experiments are rarely self-explanatory, while fully descriptive datasets demand documentation and curation effort that few labs can sustain. Coding agents, which can read code and text quickly with apparent attention to details humans tend to skim, are a natural candidate to reduce this burden. To measure how well coding agents perform data reuse, we benchmark them on eight recent papers with large-scale mouse neural recordings spanning diverse recording modalities, behavioral paradigms, and formats. Given the data, code, and paper, the agents must reformat the data for a common downstream task: training a decoder from neural activity to task and behavioral variables. Evaluating the resulting solutions, however, is challenging, since many conversion decisions are under-specified and several choices are defensible. Yet a subtle but scientifically significant error can invalidate the dataset in ways aggregate metrics can miss. We therefore pair outcome-based metrics with a process-based rubric, scored by two human evaluators against manual reference solutions. We find that agents succeed on most subtasks but rarely produce a fully error-free end-to-end solution; we characterize their mistakes and suggest data-sharing practices for agent-assisted reuse. Finally, we find that agent judges are unreliable at catching errors on their own, but can automate process-based evaluation when given reference solutions.

Machine Learning (cs.LG)

arXiv:2605.12808 [cs.LG]

(or arXiv:2605.12808v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.12808 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.