Three years of progress in 500 lines of code

TL;DR There is some consensus that LLMs are bad at hard-to-verify tasks. The question is whether models are getting better at them over time. As a motivating example, I gave one research-reproduction task to 12 models spanning three years of progress, to illustrate how (1) what looked like an emergent capability was a predictable trend, visible years earlier if progress was measured with granularity, and (2) for the earliest models, building a verifiable check would have been close to impossible: their attempts were so far from correct that a check would have had nothing to grade, so the task itself would have looked unverifiable. When thinking about current capabilities or forecasting future ones, researchers should be aware that binary judgments of usefulness can hide steady partial progress, and that a task looking 'unverifiable' today may say as much about the current capability profile of models as about the task itself. Introduction Frontier models like Fable seem to struggle with novel end-to-end research [1, 2], often producing slop, sometimes slop so bad that it would get you banned from arXiv [3]. Meanwhile, SWEs and researchers seem to find these tools incredibly useful. The best models are now capable of impressive feats of engineering, like migrating Bun to Rust in a week [4] or replicating full programs that took years to create [5]. This gap in capabilities has been documented before [6, 7, 8], and is sometimes referred to as easy- vs hard-to-verify tasks. The standard explanation is that RLVR favors progress on tasks where good and bad outputs are easy to tell apart. Especially when talking to other researchers, this gap is sometimes presented as evidence for longer timelines to an AI intern. A common version of the argument goes: Research taste has incredibly long feedback loops. Even peer review is often not enough to know how impactful a research idea will be; the only judge is time and impact in the real world. And hence "fancy tricks" with LLM judges, human preference, or simulated research universes will not suffice. We have two ways of forming an opinion about progress in these domains: we can use our own judgment, formed by working with these models on a daily basis, or we can develop targeted evaluations. Creating evals for hard-to-verify tasks is notoriously hard, and making them scalable requires tradeoffs like using imperfect LLM judges or measuring recall on human ground truth. Because of this I imagine that most of us rely on our daily experience using these models to gauge their capabilities in hard-to-verify tasks. In practice, we may ask the model to autonomously design an experiment or to come up with new research ideas. You check the output, and often it is not good enough as it is; major modifications would be needed for it to be viable as an experiment or idea. Over generations, if models keep producing ideas and experiments that fall below the acceptance threshold, it could feel like no progress is being made, even if each generation gets closer to the bar.[1] A similar phenomenon has been documented for benchmarks: what look like emergent abilities can be artifacts of the evaluation metric [9]. The underlying capability improves smoothly, while the metric stays flat and then jumps. As a motivating example, for what could be happening with hard-to-verify tasks, I give an example of a task, where models get progressively better at long before any output would look useful to someone working with the model. The Task: AI safety via debate (MNIST MCTS Debate) I've chosen a task which is both code-heavy and has a conceptual dimension to it. One of the seminal papers in AI safety research is AI Safety via Debate (Irving, Christiano & Amodei, 2018), where debate is introduced as an alignment method. In that paper, debate was tested on MNIST classification. A judge, a small CNN, is trained to classify digits from only six revealed pixels. Two debaters then play a game over an image: one argues for the true label, the other commits to a lie, and they alternately reveal pixels until the judge sees six and picks a winner. The debaters are not trained models; they play the game with Monte Carlo tree search (MCTS). A correct implementation therefore has two halves: training the sparse-pixel judge, and the MCTS debate game. I wrote a list of 14 requirements that a correct implementation has to meet. I then measured how many of them each attempt met. Requirements met (out of 14) by each attempt, ordered by model release date. Three attempts per model. GPT-3 (davinci-002): tries to implement the MNIST classifier, but gets lost quickly; the code looks straight out of sleep paralysis. GPT-3.5: much better formatting, and it defines the MNIST classifier correctly, but it either fails to train on sparse pixels, uses random play, or has other conceptual failures. GPT-4: still implements neither the judge nor the debate game correctly. GPT-4 Turbo: the first model that extensively uses