analiza aiqbench
EvoClaw exposes a critical gap in current AI agent capabilities, revealing that performance drops from over 80% on isolated tasks to under 38% when required to maintain long-term software integrity. This benchmark highlights the urgent need for evaluating agents on temporal dependencies and error accumulation rather than one-off coding tasks. The findings suggest that existing models are ill-equipped for the autonomous, continuous evolution necessary in real-world dynamic environments.
EvoClaw: Evaluating AI Agents on Continuous Software Evolution
arXiv:2603.13428v2 Announce Type: replace-cross Abstract: With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
Abstract:With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable EvoClaw, a novel benchmark that requires agents to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
arXiv:2603.13428 [cs.SE]
(or arXiv:2603.13428v2 [cs.SE] for this version)
Text integral sub licența deschisă a publicației · 159 cuvinte
Subiecte conexe
SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
arXiv · 23 iul. 2026
AI and Quantum Computing Briefing
livetradingnews.com · 4 aug. 2026
Import AI 464: Fables writes GPU kernels; AI automation; and analog computation
Import AI · 6 iul. 2026
EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
arXiv · 12 iun. 2026