analiza aiqbench
This benchmark evaluates the capability of coding agents to handle real-world maintenance tasks specified natively in Russian using authentic open-source repositories. The study highlights critical product-level behaviors, such as silent model substitution and oracle-hunting artifacts, which distort performance metrics beyond simple language proficiency. By withholding regression test oracles until after evaluation, this work establishes a rigorous standard for measuring the actual efficacy of AI agents in non-English development environments.
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
arXiv:2607.06411v2 Announce Type: replace-cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify), each specified natively in Russian -- written from scratch, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All fix commits postdate the training-data cutoffs of every evaluated model. Round 1 evaluates Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 (3 independent runs each; pass@1 with task-level uncertainty); the best configuration resolves 78.7% of tasks. Auditing full trajectories of an hors-concours configuration (Claude Code + Fable 5), we caught the product silently substituting the model on 20% of tasks via an official safeguard fallback -- evidence that the deployed product, not the model, is the unit actually measured. Version 2 adds Round 2: seven further configurations on the same frozen set under a per-configuration freshness gate -- the Russian-market agents SourceCraft CLI (ds, legacy) and Koda CLI (koda-pro), Antigravity with Gemini 3.1 Pro and 3.5 Flash, and Codex CLI with GPT-5.6 Sol and Luna. SourceCraft's flagship resolves 68.1% (N=23), above GPT-5.5 and both Gemini rows. A tool-call contamination re-audit of all 437 Round-2 trajectories finds the Russian and Gemini columns clean (0/293 cells) while flagging systematic oracle-hunting in the GPT-5.6 family (8/69 and 13/75 cells), including one case of mining a prior round's artifacts from the run machine's disk; honest scores are published alongside raw ones. We release statements, metadata, trajectories, and diffs; oracles are withheld with a SHA-256 manifest.
Abstract:Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify), each specified natively in Russian -- written from scratch, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All fix commits postdate the training-data cutoffs of every evaluated model. Round 1 evaluates Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 (3 independent runs each; pass@1 with task-level uncertainty); the best configuration resolves 78.7% of tasks. Auditing full trajectories of an hors-concours configuration (Claude Code + Fable 5), we caught the product silently substituting the model on 20% of tasks via an official safeguard fallback -- evidence that the deployed product, not the model, is the unit actually measured. Version 2 adds Round 2: seven further configurations on the same frozen set under a per-configuration freshness gate -- the Russian-market agents SourceCraft CLI (ds, legacy) and Koda CLI (koda-pro), Antigravity with Gemini 3.1 Pro and 3.5 Flash, and Codex CLI with GPT-5.6 Sol and Luna. SourceCraft's flagship resolves 68.1% (N=23), above GPT-5.5 and both Gemini rows. A tool-call contamination re-audit of all 437 Round-2 trajectories finds the Russian and Gemini columns clean (0/293 cells) while flagging systematic oracle-hunting in the GPT-5.6 family (8/69 and 13/75 cells), including one case of mining a prior round's artifacts from the run machine's disk; honest scores are published alongside raw ones. We release statements, metadata, trajectories, and diffs; oracles are withheld with a SHA-256 manifest.
20 pages, 1 figure, 9 tables. v2 adds Round 2: Russian-market coding agents (SourceCraft CLI, Koda CLI), Antigravity with Gemini 3.1 Pro / 3.5 Flash, and Codex CLI with GPT-5.6 on the same frozen task set, plus a tool-call contamination re-audit (network + disk layers). Data, full trajectories and harness: this https URL
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2607.06411 [cs.SE]
(or arXiv:2607.06411v2 [cs.SE] for this version)
Text integral sub licența deschisă a publicației · 360 cuvinte
Subiecte conexe
Large-scale Repository Engineering via Agent-Native Reusable Code Primitives
arXiv · 8 oct. 2026
Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM
AWS · 9 sept. 2026
News Archive 2026
Quantum Computing Report · 25 iul. 2026
Introducing North Mini Code: Cohere’s First Model For Developers
Hugging Face · 9 iun. 2026