analiza aiqbench
The paper argues that reinforcement learning with only unit-test rewards teaches code models to make larger edits and to invent bugs in correct code, even as pass rates rise. Its proposed reward adds a signal for semantically verified, localized edits, which cuts average edit size from 7-15 lines to about 3 without hurting pass rates. For readers, the main open question is whether these gains hold beyond the reported benchmarks and model sizes, since the results come from one group's experiments.
PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards
arXiv:2610.18755v1 Announce Type: cross Abstract: Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this unit-test-only reward teaches a model during debugging and find that, while it raises pass rates, it also teaches the model to edit more and to hallucinate bugs in code that is already correct. We introduce PreciseCoder, a training algorithm that augments the unit-test reward with a signal for semantically verified, localized edits, which correlates with smaller edits and higher pass rates. On 4B, 9B, and 27B models, PreciseCoder reduces the average number of edit lines from 7-15to 3 at a pass rate on par with unit-test-only training. Trained only on single-line bugs, PreciseCoder can generalize to repository-level program repair, where it resolves more issues with smaller patches. Its precision also transfers from weaker to stronger models: given the reasoning trace of a 9B PreciseCoder model, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gain 4.6-12.4 points in edit precision with no additional training. Linguistic analysis of the thinking traces shows that our reward encourages the model to reason about individual lines, name the buggy ones, and form consistent bug hypotheses.
Abstract:Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this unit-test-only reward teaches a model during debugging and find that, while it raises pass rates, it also teaches the model to edit more and to hallucinate bugs in code that is already correct. We introduce PreciseCoder, a training algorithm that augments the unit-test reward with a signal for semantically verified, localized edits, which correlates with smaller edits and higher pass rates. On 4B, 9B, and 27B models, PreciseCoder reduces the average number of edit lines from 7-15to 3 at a pass rate on par with unit-test-only training. Trained only on single-line bugs, PreciseCoder can generalize to repository-level program repair, where it resolves more issues with smaller patches. Its precision also transfers from weaker to stronger models: given the reasoning trace of a 9B PreciseCoder model, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gain 4.6-12.4 points in edit precision with no additional training. Linguistic analysis of the thinking traces shows that our reward encourages the model to reason about individual lines, name the buggy ones, and form consistent bug hypotheses.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.18755 [cs.SE]
(or arXiv:2610.18755v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2610.18755 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 300 cuvinte
Subiecte conexe
Import AI 470: No rights for machines; automating environment generation with SPADE; and building better GPU kernels with Hawkeye
Import AI · 24 aug. 2026
Benchtalks #2: The future of coding benchmarks
Snorkel AI Blog · 3 iun. 2026
GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior
LessWrong · 15 oct. 2026
[AINews] not much happened today
Latent Space · 3 oct. 2026