Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Title:Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Abstract:A widespread practice in software development is to tailor coding agents to repositories using context files, such as this http URL. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files, and a novel collection of issues from repositories containing developer-committed context files. Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files. Specifically, we find that while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful. We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI)

arXiv:2602.11988 [cs.SE]

(or arXiv:2602.11988v3 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2602.11988 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.