aiqbench analysis
Researchers have identified significant instability in quantum software benchmarks, revealing that nearly half of reported Qiskit defects are artifacts caused by environment inconsistencies rather than actual code errors. This finding underscores the critical necessity of version-pinned validation protocols to ensure reliable evaluation before testing AI models' repair capabilities on real-world quantum issues. Consequently, assessing LLM performance requires rigorous benchmark integrity checks specific to each software release pair.
GPT-5.4 Repairs 48.8% of Qiskit Bugs Across Six Versions
Researchers tested LLMs on real Qiskit bugs, finding GPT-5.4 successfully repaired 48.8% of defects across six versions—outperforming other models.
GPT-5.4 successfully repaired 48.8% of bugs across six versions of Qiskit, a leading quantum software development kit, achieving the highest pass@10 metric among the models tested, including GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). Researchers, led by Saumya Brahmbhatt, evaluated the LLMs using Bugs4Q, a benchmark of 67 unique real Qiskit defects, and discovered an issue with benchmark reliability: 64% of successful repairs addressed code already flagged as invalid by the quantum software environment. This finding highlights that correctness depends on the specific pairing of benchmark and Qiskit version, rather than the benchmark alone. The team’s work introduces a version-pinned validation protocol and a re-validated Bugs4Q benchmark, demonstrating that benchmark validation must precede repair evaluation. Nearly half of attempted quantum code repairs using advanced artificial intelligence are, in effect, fixing code that was already broken by the testing environment itself, according to new research into the reliability of quantum…
Excerpt shown under the publisher's RSS terms · 149 words · full text and figures are on the publisher's site
Related coverage
Fault-Class-Matched Test Oracles for Output-Invisible Quantum Transpiler Regressions
arXiv · Sep 22, 2026
How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair
arXiv · Jul 29, 2026
An Executable Benchmark for LLM-Based HLS Repair:Design Complexity and Repair Underconstraint
arXiv · Oct 7, 2026
SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
arXiv · Oct 12, 2026