analiza aiqbench
QCircuitEval introduces a critical advancement in evaluating LLM-generated quantum code by shifting from output matching to verifying deterministic behavioral contracts. This approach resolves a fundamental flaw in current benchmarks where syntactically different but functionally correct circuits are unfairly penalized due to reliance on single reference solutions. By supporting major frameworks like Qiskit, Cirq, and PennyLane, the tool establishes a more robust standard for assessing quantum software development capabilities.
QCircuitEval leverages quantum frameworks—Qiskit, Cirq, PennyLane, CUDA-Q.
QCircuitEval is an open-source, deterministic behavior-contract benchmark for large language model (LLM)-generated quantum circuit code across Qiskit, Cirq, PennyLane, and CUDA-Q. QCircuitEval grades a program by what it does, not just by what it outputs, utilizing both a structural and a functional grader.
A single, stored solution may unfairly penalize correct quantum code, according to researchers behind QCircuitEval, a new benchmark for large language model-generated quantum circuits. While benchmarks like Qiskit HumanEval and QuanBench+ currently rely on comparing outputs to one example, QCircuitEval instead assesses what a program does, not just what it outputs, across frameworks including Qiskit, Cirq, and PennyLane. "Matching an expected output is not proof that a quantum program performed the requested computation," the team explains, emphasizing the need to verify both functional behavior and output. QCircuitEval employs both structural and functional graders, ensuring a deterministic verdict based on a contract derived from the task itself, and pinned…
Extras afișat conform termenilor RSS ai publicației · 108 cuvinte · textul integral și figurile sunt pe site-ul publicației
