FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration

Abstract:AI coding assistants are rapidly becoming integral to modern software development. A key challenge in this space is the continual need to migrate and modernize codebases in response to evolving software ecosystems. Traditionally, such migrations have relied on rule-based systems and human intervention. With the advent of powerful large language models (LLMs), AI-driven agentic frameworks offer a promising alternative-but their effectiveness has not been systematically evaluated. In this paper, we introduce \FreshBrew{}\footnote{\url{this https URL}}, a novel benchmark for evaluating AI agents on project-level Java migrations, with a specific focus on measuring an agent's ability to preserve program semantics and avoid reward hacking, which we argue requires projects with high test coverage for a rigorous and reliable evaluation. We benchmark several state-of-the-art LLMs, and compare their performance against established rule-based tools. Our evaluation on a challenging subset of 228 public Maven repositories that build on JDK 8, fail on JDK 17, and have high test coverage shows that the top-performing model, Gemini 2.5 Flash, can successfully migrate 52.3\% of projects to JDK 17. Our empirical analysis reveals novel insights into the critical strengths and limitations of current agentic approaches, offering actionable insights into their real-world applicability. Our empirical study reveals failure modes of current AI agents in realistic Java modernization tasks, providing a foundation for evaluating trustworthy code-migration systems. By releasing \FreshBrew{}, we aim to facilitate rigorous, reproducible evaluation and catalyze progress in AI-driven codebase modernization.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI)

arXiv:2510.04852 [cs.SE]

(or arXiv:2510.04852v3 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2510.04852 Focus to learn more arXiv-issued DOI via DataCite

https://doi.org/10.1145/3744916.3773214 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.