analiza aiqbench
This paper is about training software engineering agents with reinforcement learning, so the primary subject is ML, and it falls squarely in the Machine Learning category. The core contribution is a reward-design change: CodeLift reweights test pass-rate rewards by how many patch groups pass each test group, so harder tests carry more weight than redundant trivial ones. Reported gains over binary and pass-rate baselines on SWE benchmarks are promising, but readers should check how robust they are across benchmarks, base models, and compute budgets before treating the method as a general improvement.
CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum
arXiv:2610.17224v1 Announce Type: new Abstract: The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.
Abstract:The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.
Machine Learning (cs.LG); Software Engineering (cs.SE)
arXiv:2610.17224 [cs.LG]
(or arXiv:2610.17224v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.17224 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 288 cuvinte
Subiecte conexe
From Foundational Competency to Expert Performance: A Curriculum Approach to Model Development
Snorkel AI Blog · 15 sept. 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · 15 aug. 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
Introducing Gemma 4 models on Amazon Bedrock
AWS · 15 iun. 2026