analiza aiqbench
Tencent WorkBuddy Bench introduces a novel methodology for evaluating coding agents by reverse-engineering real-world commits into contamination-resistant prompts across four distinct domains, including security workflows. By releasing the full evaluation harness and reference solutions openly while preventing prompt leakage via web search, this benchmark aims to provide an auditable metric of agent performance that avoids common data poisoning issues found in public datasets. The framework's focus on reproducible testing protocols for both CodeBuddy and Claude models establishes a new standard for validating practical coding capabilities beyond simple code generation tasks.
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
arXiv:2607.20911v1 Announce Type: new Abstract: We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running distribution-informed coding-agent tasks across four work domains - Code, Web, Office, and Security. Rather than adapting public issue text, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. Because the dataset is released openly - task directories, environment images, evaluation harness, tests, and reference solutions - contamination resistance rests on this construction together with dataset versioning rather than on secrecy. The four subsets - repository-level engineering, front-end development, office and business workflows, and red-/blue-team security - probe complementary facets of real work, each with its own verification style. All are packaged in a uniform task-directory format and run, under a uniform and reproducible protocol, on two agent harnesses (CodeBuddy Code and Claude Code); the full open release makes the benchmark reproducible end to end and directly auditable, since any third party can re-run each task and inspect its content. Because each subset uses a different scoring instrument, scores are not comparable across subsets and the suite reports no suite-wide average. We report a cross-model leaderboard across several model families.
Abstract:We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running distribution-informed coding-agent tasks across four work domains - Code, Web, Office, and Security. Rather than adapting public issue text, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. Because the dataset is released openly - task directories, environment images, evaluation harness, tests, and reference solutions - contamination resistance rests on this construction together with dataset versioning rather than on secrecy. The four subsets - repository-level engineering, front-end development, office and business workflows, and red-/blue-team security - probe complementary facets of real work, each with its own verification style. All are packaged in a uniform task-directory format and run, under a uniform and reproducible protocol, on two agent harnesses (CodeBuddy Code and Claude Code); the full open release makes the benchmark reproducible end to end and directly auditable, since any third party can re-run each task and inspect its content. Because each subset uses a different scoring instrument, scores are not comparable across subsets and the suite reports no suite-wide average. We report a cross-model leaderboard across several model families.
30 pages, 9 figures. Project page: this https URL ; code: this https URL ; dataset: this https URL
Computation and Language (cs.CL); Software Engineering (cs.SE)
arXiv:2607.20911 [cs.CL]
(or arXiv:2607.20911v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.20911 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 276 cuvinte
Subiecte conexe
[AINews] High Taste Testers Ep. 1
Latent Space · 17 oct. 2026
Academia is for Ambition — Alex Zhang, MIT
Latent Space · 2 oct. 2026
E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
arXiv · 28 iul. 2026
Senior SWE-Bench: Evaluating Coding Agents Like Senior Engineers
Snorkel AI Blog · 16 iul. 2026