aiqbench analysis
This research introduces a novel framework for agentic synthesis that uses counterexample-supplemented sketches to enforce policy integrity during code generation, addressing critical safety issues where agents inadvertently discard domain rules. By requiring operator approval of corrected behaviors and maintaining deterministic output comparisons against curated regression sets, the method effectively prevents drift while preserving provenance in dynamic implementation environments.
Agentic Synthesis against Counterexample-Supplemented Sketches
arXiv:2607.15854v2 Announce Type: replace-cross Abstract: Coding agents can fix a failing example without preserving the domain rule that made it fail. We present agentic synthesis against counterexample-supplemented sketches, a repository-native method for systems whose policy is discovered during implementation. A human starts with a partial sketch, and a coding agent compiles a replaceable projection. When simulation exposes missing or mistaken policy, an operator approves the corrected behavior and the minimum general rule the case authorizes. Every Developer call names its change authority and the rules, holes, anchors, and approved behavior that must survive. Conflict or ambiguous permission leaves the files unchanged and produces a clarification question. A complete archive preserves provenance; a curated regression set gates distinct boundaries. Before another candidate is revealed, the active case and curated regressions must pass both deterministic approved-output comparison and a separate review against the current sketch. Periodic clean regeneration tests whether the sketch carries the learned policy. We demonstrate the method with CatSynth, a captured synthetic application. In one open-world run with GPT-5.4-mini, 8 of 14 frozen candidates became counterexamples. Under the corrected protocol, replay-all, evolved-sketch rebuild, and retained Sketch-CE each passed all 8 accepted cases. They passed 14, 17, and 16 of 21 withheld cases, respectively. Sketch review rejected premature empty-input and tag policies and restored dropped anchors; adjudicated reviewer errors did not become policy. One model and one reveal order cannot establish general correctness or superiority. On this suite, the second check exposed drift hidden by deterministic replay, and the reviewed sketch passed three more withheld cases than raw example replay.
Abstract:Coding agents can fix a failing example without preserving the domain rule that made it fail. We present agentic synthesis against counterexample-supplemented sketches, a repository-native method for systems whose policy is discovered during implementation. A human starts with a partial sketch, and a coding agent compiles a replaceable projection. When simulation exposes missing or mistaken policy, an operator approves the corrected behavior and the minimum general rule the case authorizes. Every Developer call names its change authority and the rules, holes, anchors, and approved behavior that must survive. Conflict or ambiguous permission leaves the files unchanged and produces a clarification question. A complete archive preserves provenance; a curated regression set gates distinct boundaries. Before another candidate is revealed, the active case and curated regressions must pass both deterministic approved-output comparison and a separate review against the current sketch. Periodic clean regeneration tests whether the sketch carries the learned policy. We demonstrate the method with CatSynth, a captured synthetic application. In one open-world run with GPT-5.4-mini, 8 of 14 frozen candidates became counterexamples. Under the corrected protocol, replay-all, evolved-sketch rebuild, and retained Sketch-CE each passed all 8 accepted cases. They passed 14, 17, and 16 of 21 withheld cases, respectively. Sketch review rejected premature empty-input and tag policies and restored dropped anchors; adjudicated reviewer errors did not become policy. One model and one reveal order cannot establish general correctness or superiority. On this suite, the second check exposed drift hidden by deterministic replay, and the reviewed sketch passed three more withheld cases than raw example replay.
32 pages, 5 displayed figures (4 distinct screenshots). Includes the CatSynth artifact supplement. Code and captured experiment artifacts: this https URL Clarifies the two-check CESS method and Developer change authority; adds the protocol-correct CatSynth rerun and replaces the prior withheld-case headline
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2607.15854 [cs.SE]
(or arXiv:2607.15854v2 [cs.SE] for this version)
Full text under the publisher's open licence · 310 words
Related coverage
Agentic Synthesis against Counterexample-Supplemented Sketches
arXiv · Jul 20, 2026
What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review
arXiv · Sep 29, 2026
The Alignment Journal: Organization, Personnel, and Scope
AI Alignment Forum · Sep 1, 2026
Counterexample Guided Learning in the Large using Reasoning Agents
arXiv · Jun 12, 2026