Agents’ Last Exam: AI Benchmarking for Real Work

At our latest Snorkel AI Reading Group, Yiyou Sun and David (Xinyang) Han (UC Berkeley, Center for Responsible and Decentralized Intelligence) presented Agents’ Last Exam (ALE) — a benchmark designed to evaluate AI agents on long-horizon, economically valuable, real-world tasks with verifiable outcomes. ALE is a collaboration between Berkeley RDI, Snorkel AI, and 300+ expert contributors across 55 professional subfields. ALE asks a deceptively simple question: can today’s AI agents actually do the work that matters economically? The answer so far — with the hardest tier averaging below 1% full pass rate across frontier models — suggests we’re further from that bar than leaderboards imply. Lightly edited for readability. Thank you, Jason. I appreciate that Snorkel provided this opportunity to share our recent work. If you have questions, I’m happy to connect after the presentation as well. So we call this Agents’ Last Exam. People know “Humanity’s Last Exam” — we borrowed the name. I’m actually a little regretful about that. I think it should have been called Agent’s First Exam, because I feel it’s a first genuine attempt to benchmark real workflows that experts actually perform. It’s definitely not the last. If I had another chance, I’d use “first.”…