aiqbench analysis
This arXiv paper studies how coding agents can offload routine repair decisions to a small probabilistic judge to cut token costs. Its main finding is that savings depend on which work an intervention changes, not just on what the judgment itself costs. The reported 21.6%-50.6% token reduction on SWE-bench tasks, with no drop in repair rates, is a concrete efficiency result for teams running agentic coding workflows, though it comes from a single arXiv preprint and has not been independently replicated.
JevCoder: When Does Decision Delegation Pay Off in Code Repair?
arXiv:2610.19596v1 Announce Type: cross Abstract: A coding agent repeatedly makes decisions as it repairs code. These decisions shape repair trajectories and affect the token cost of successive model requests. We ask which repair decisions can be delegated to a low-cost probabilistic judge to reduce token use while maintaining repair quality. The judges answer questions with probabilities over predefined answers, without generating explanations. We evaluate judgment quality across repair stages and test how individual interventions change subsequent repair work. We compare these interventions with unguided repair, which uses no judge, and with the same workflow using the repair model itself as the judge. The benefit depends on the work an intervention changes, rather than the cost of the judgment alone. Based on these studies, we propose JevCoder, a workflow built around Jev, a low-cost probabilistic judge. It combines code-snippet recommendations with action-type guidance and uses judgments to determine when to submit. We evaluate three repair models, each on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Compared to unguided repair, JevCoder reduces mean repair-model token use by 21.6%-50.6%, with higher observed repair rates across all three models.
Title:JevCoder: When Does Decision Delegation Pay Off in Code Repair?
Abstract:A coding agent repeatedly makes decisions as it repairs code. These decisions shape repair trajectories and affect the token cost of successive model requests. We ask which repair decisions can be delegated to a low-cost probabilistic judge to reduce token use while maintaining repair quality. The judges answer questions with probabilities over predefined answers, without generating explanations. We evaluate judgment quality across repair stages and test how individual interventions change subsequent repair work. We compare these interventions with unguided repair, which uses no judge, and with the same workflow using the repair model itself as the judge. The benefit depends on the work an intervention changes, rather than the cost of the judgment alone. Based on these studies, we propose JevCoder, a workflow built around Jev, a low-cost probabilistic judge. It combines code-snippet recommendations with action-type guidance and uses judgments to determine when to submit. We evaluate three repair models, each on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Compared to unguided repair, JevCoder reduces mean repair-model token use by 21.6%-50.6%, with higher observed repair rates across all three models.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2610.19596 [cs.SE]
(or arXiv:2610.19596v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2610.19596 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 294 words
Related coverage
RAC: A Relational Agent Contract for Constraint-Governed Engineering Tasks
arXiv · Oct 17, 2026
Welcome Beam: a 501B open model you can own, customize, and scale
Hugging Face · Oct 16, 2026
Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy
arXiv · Aug 28, 2026
Cognition's Jeff Wang on demoting the IDE, with Richard Liu from Anthropic
1Password · Aug 13, 2026