aiqbench analysis
This paper introduces a novel, cost-effective method for generating high-quality coding benchmarks by programmatically composing verified atomic problems without relying on large language models for synthesis or validation. The resulting tasks effectively challenge frontier AI models and serve as valuable training data to improve performance through rejection-sampling fine-tuning. This approach addresses the critical bottleneck of expensive human curation in developing robust evaluation suites for software engineering capabilities.
CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph
arXiv:2610.15345v1 Announce Type: cross Abstract: High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typically rely on costly model generation and validation. In this work, we show that new coding problems can be created at almost no cost by composing existing ones. We formulate problem generation as code composition: verified atomic problems are combined into level-$N$ tasks, and a task is kept only if every atomic problem remains unsolved and a joint reference solution passes the same check used at evaluation time. The entire task-construction process is programmatic and requires no language-model generation or validation. We instantiate this approach in four coding settings: multi-bug repair, proof completion, complexity analysis, and clone equivalence. The generated tasks are challenging for frontier models, which all degrade as compositional depth increases. The generated tasks are also useful as training data: rejection-sampling fine-tuning on them improves Qwen3.5-9B at every composition level.
Abstract:High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typically rely on costly model generation and validation. In this work, we show that new coding problems can be created at almost no cost by composing existing ones. We formulate problem generation as code composition: verified atomic problems are combined into level-$N$ tasks, and a task is kept only if every atomic problem remains unsolved and a joint reference solution passes the same check used at evaluation time. The entire task-construction process is programmatic and requires no language-model generation or validation. We instantiate this approach in four coding settings: multi-bug repair, proof completion, complexity analysis, and clone equivalence. The generated tasks are challenging for frontier models, which all degrade as compositional depth increases. The generated tasks are also useful as training data: rejection-sampling fine-tuning on them improves Qwen3.5-9B at every composition level.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2610.15345 [cs.LG]
(or arXiv:2610.15345v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.15345 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 248 words
Related coverage
Links #5: 2026/07
LessWrong · Jul 31, 2026
From Vector Search to Agentic RAG: Building an Enterprise Research Analyst with hipVS
AMD · Jul 15, 2026
Multi-agent social intelligence with Strands Agents and Amazon Bedrock
AWS · Jul 14, 2026
Presentation: Can Claude Fix Itself? Using LLMs for Incident Response
InfoQ · Aug 26, 2026