OSWorld 2.0: Why Long-Horizon Computer-Use Agents Still Fail Four Out of Five Tasks

At our latest Snorkel AI Reading Group, held virtually for the first time, Mengqi Yuan (XLANG Lab, University of Hong Kong) presented OSWorld 2.0, a benchmark of 108 long-horizon, real-world computer-use workflows spanning 31 self-hosted websites and professional desktop applications. Tasks average more than 300 agent steps, 69.6% take a skilled human over an hour to finish, and the best system in the paper still completes only 20.6% of them outright. OSWorld 2.0 was developed at XLANG Lab with support from Snorkel AI’s Open Benchmarks Grants program, and it was one of the benchmarks Anthropic used to evaluate Claude Fable 5.1 when that model launched less than a day before this talk. The question Yuan’s team set out to answer: once a benchmark stops looking like a short, clean scripted task and starts looking like an actual afternoon of work, how far do today’s frontier agents actually get? Lightly edited for readability. Why OSWorld 2.0: OSWorld 1.0 was saturating Yuan opened by explaining the motivation. OSWorld 1.0, the team’s original benchmark, evaluates computer-use agents on real-world tasks. It had done its job: agent scores climbed for two years, and by earlier this year the best systems were clearing 80% accuracy.…