Senior SWE-Bench: Evaluating Coding Agents Like Senior Engineers

At our latest Snorkel AI Reading Group, Henry Ehrenberg presented Senior SWE-Bench, an open-source, Harbor-compatible benchmark for evaluating coding agents on realistic, senior-level software engineering work. Its 100 tasks, with 50 public and 50 kept private to mitigate contamination, are sourced from real pull requests across 12 production repositories and cover complex features, migrations, bugs, and performance issues. Senior SWE-Bench was developed by Snorkel AI in collaboration with researchers from Princeton and UW–Madison. Explore the leaderboard. Unlike benchmarks that provide detailed technical specifications, Senior SWE-Bench gives agents the kinds of requests engineers actually receive, describing the desired outcome without prescribing the solution. To evaluate open-ended feature tasks reliably, it combines pre-written verifiers with a validation agent that uses expert-designed recipes to write behavioral tests adapted to submitted solutions. A separate taste judge scores minimality, approach quality, hygiene, fluency, and craftsmanship relative to the existing codebase and reference solution. Get the scoop on new benchmarks, research, and exclusive events. By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy. The results show how far coding…