Benchtalks #2: The future of coding benchmarks

For our second Benchtalks, the series dedicated to the researchers building the measurement toolkits that frontier labs hill-climb on, Snorkel AI co-founder Vincent Sunn Chen sat down with John Yang, a Stanford PhD student and creator of the SWE-bench franchise, SWE-smith, CodeClash, and most recently ProgramBench. Every frontier model scored 0% at launch: ProgramBench tests whether language models can rebuild real software (FFmpeg, SQLite, the PHP interpreter, the tinycc compiler) from scratch with no internet access. At launch, every frontier model on the leaderboard scored 0%. A week later, GPT 5.5 became the first model to crack a single task. As Yang puts it, “I can’t implement FFmpeg from scratch in one month. But maybe the model can. That’s a gear shift that’s meaningful.” Cheating detection is its own benchmark problem: When the team allowed models internet access during evaluation, up to 36% of strong-model runs were flagged by a 9-judge LM panel for looking up source code, despite an explicit system prompt forbidding it. Those judges disagreed 40 to 57 percent of the time on what counted as cheating. Yang’s stance: “Rigor is king, reliability is king, reproducibility is king. Everything else is cherry on top.” Verification has evolved…