analiza aiqbench
This paper benchmarks whether frontier language agents can repair real ROS 2 navigation bugs, grading patches by simulated robot behavior rather than unit tests. Repair rates of 24-36% and a large gap between oracle types (67% vs. 22%) suggest that evaluation design, not just model capability, drives reported results. For AI readers, the most useful part is the adversarial audit: a simple pass/fail rule accepted "counterfeit" patches that suppressed symptoms without fixing defects, and the authors argue better discrimination requires richer observation rather than stricter thresholds.
RoboBugBench: Benchmarking Automated Repair of Real Robot Software Under an Audited Behavioral Oracle
arXiv:2610.16369v1 Announce Type: new Abstract: Language agents increasingly modify robot software, but whether they can repair it is unmeasured, because the domain supplies no oracle to inherit: a robot bug manifests as behavior, and robot repositories ship almost no tests that capture it. We present RoboBugBench, a benchmark of 33 real, execution-validated defects in a pinned ROS 2 navigation stack, in which a candidate patch is graded by compiling it and driving the robot through a physics simulation rather than by running a unit test. Three frontier models, each given 1 attempt per instance, repair 24.2%-36.4%, with no pairwise difference significant; difficulty tracks the oracle rather than the domain, at 66.7% on instances graded by process survival against 22.2% on those graded by what the robot did (instance-level p = 0.0027). Because the oracle is constructed rather than inherited we audit it adversarially, and report the decomposition that audit yields: counterfeit patches which suppress a failure without repairing the defect are accepted by the shipped rule 11/11, while a distributional rule over the same data rejects 4 at no cost in false rejects. The residual is an observability bound (what the observation channel discards, no predicate over it can recover), so discrimination must be bought with instrumentation rather than stricter criteria, and our repair rates are bounds: inflated by what the oracle cannot distinguish, deflated by unanimity's rejection of correct fixes under run-to-run noise. We release the corpus, criterion batteries, counterfeit suite and cached telemetry.
Abstract:Language agents increasingly modify robot software, but whether they can repair it is unmeasured, because the domain supplies no oracle to inherit: a robot bug manifests as behavior, and robot repositories ship almost no tests that capture it. We present RoboBugBench, a benchmark of 33 real, execution-validated defects in a pinned ROS 2 navigation stack, in which a candidate patch is graded by compiling it and driving the robot through a physics simulation rather than by running a unit test. Three frontier models, each given 1 attempt per instance, repair 24.2%-36.4%, with no pairwise difference significant; difficulty tracks the oracle rather than the domain, at 66.7% on instances graded by process survival against 22.2% on those graded by what the robot did (instance-level p = 0.0027). Because the oracle is constructed rather than inherited we audit it adversarially, and report the decomposition that audit yields: counterfeit patches which suppress a failure without repairing the defect are accepted by the shipped rule 11/11, while a distributional rule over the same data rejects 4 at no cost in false rejects. The residual is an observability bound (what the observation channel discards, no predicate over it can recover), so discrimination must be bought with instrumentation rather than stricter criteria, and our repair rates are bounds: inflated by what the oracle cannot distinguish, deflated by unanimity's rejection of correct fixes under run-to-run noise. We release the corpus, criterion batteries, counterfeit suite and cached telemetry.
14 pages: 9 main + references + two appendices. 1 figure, 4 tables. Accepted non-archival workshop paper at the Verify-Agents workshop ('Who Verifies the Agents?') at NeurIPS 2026, Sydney, Dec 11-12, 2026
Robotics (cs.RO)
arXiv:2610.16369 [cs.RO]
(or arXiv:2610.16369v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2610.16369 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 289 cuvinte
