aiqbench analysis
MEnvAgent addresses a critical bottleneck in LLM-based software engineering by automating the construction of verifiable, multi-language execution environments, thereby enabling more robust evaluation of coding agents. The framework achieves significant efficiency gains through an environment reuse mechanism that reduces computational overhead by 43% while improving task completion rates. By releasing the largest open-source polyglot dataset of realistic Docker environments to date, this work provides a foundational resource for advancing scalable and verifiable software development benchmarks.
MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering
arXiv:2601.22859v3 Announce Type: replace-cross Abstract: The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.
Abstract:The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at this https URL.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2601.22859 [cs.SE]
(or arXiv:2601.22859v3 [cs.SE] for this version)
Full text under the publisher's open licence · 174 words
Related coverage
Graph-Guided Repository Environment Construction
arXiv · Sep 29, 2026
AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale
arXiv · Aug 24, 2026
From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents
arXiv · Oct 15, 2026
CoDA: Co-Evolution of Data Engines and Action Policies
arXiv · Oct 17, 2026