AI Safety & Evaluation Engineer

What an evaluation actually is

An "eval" is not a number. It is three things that only make sense together: a distribution of tasks you care about, a scoring rule that turns a model's behaviour on those tasks into a value, and a decision the value is supposed to inform. Change any one of the three and the same score means something different. A model that gets 90% on a set of multiple-choice chemistry questions tells you something about multiple-choice chemistry questions written in that style; whether that says anything about helping a lab technician depends entirely on how close those tasks are to what a lab technician does.

Most published benchmarks make the first two parts explicit and leave the third implicit. That is where the trouble starts. When a leaderboard score becomes a proxy for "is this model good?", the score gets optimised, and the thing it was a proxy for drifts away.

Three ways a benchmark stops meaning anything

Contamination. Models are trained on enormous web crawls. If the benchmark questions — or close paraphrases, or discussions of the answers — appear in the training data, a high score measures memory, not capability. This is hard to rule out for closed models because nobody outside the lab can inspect the data, which is the point of Leak, Cheat, Repeat (Balloccu et al.). The GSM1k paper in this module's materials demonstrates the effect cleanly: the authors wrote a fresh set of grade-school maths problems matched in style and difficulty to GSM8K, and found several models scored noticeably lower on the new set than on the old one. A gap like that is the fingerprint of contamination (or of overfitting to the benchmark's quirks, which for your purposes is the same disease).

Saturation. Once the best models score near the ceiling, the benchmark cannot separate them, and the remaining errors are often label noise rather than model failures. MMLU is the standard example: it was a useful yardstick in 2021 and is close to saturated for frontier models today, with a non-trivial fraction of its questions known to have wrong or ambiguous keys. A saturated benchmark still gets reported because the number looks good.

Goodhart's law. "When a measure becomes a target, it ceases to be a good measure." Labs tune on public benchmarks, sometimes deliberately, sometimes just by selecting checkpoints that score well. The benchmark's task distribution is then no longer a random sample of the thing you cared about; it is the thing the model was shaped for. HELM (Liang et al.) was partly a response to this: evaluate many scenarios and many metrics at once, so that optimising one does not hide regressions on another.

A worked example: reading a reported score

Suppose a release note says: "Model X scores 92.1% on HumanEval." Work through it:

1. What is the task distribution? 164 hand-written Python functions with docstrings and unit tests, written in 2021. Short, self-contained, no external libraries. That is a narrow slice of programming.
2. What is the scoring rule? pass@1: did the first sample pass all the tests? Sampling temperature and the number of attempts materially change this, and release notes often do not say what they used.
3. Could it be contaminated? HumanEval has been public on GitHub since 2021 and discussed everywhere. Assume yes unless the lab shows otherwise.
4. Is it saturated? The top models are above 90%, so the remaining 8% is a mix of genuinely hard problems and ambiguous tests. Differences of two points are not meaningful.
5. What decision does it support? "This model can write short, self-contained functions from a clear spec." It does not support "this model can work in your 200k-line codebase."

Nothing above says the score is useless. It says the score answers a narrower question than the headline implies, and your job is to say exactly which question.

What evaluators do differently

The field's answer to all three failure modes is the same move: hold out something the model cannot have seen, and keep holding out. Private test sets, freshly written items, items dated after the model's training cutoff, and periodic re-authoring are the basic hygiene. Anthropic's "Challenges in evaluating AI systems" is candid about how much engineering effort goes into just running an eval correctly: formatting choices, few-shot examples and parsing rules can move a score by several points, which is often larger than the gap between models being compared.

Two things are still unsettled. First, there is no agreed-upon test for contamination in closed models; n-gram overlap checks are weak, and labs' own statements are unverifiable. Second, the field has not converged on how to report uncertainty, which is the subject of the next module.

What to remember

• An eval is a task distribution, a scoring rule and a decision. Always name all three before quoting a number.
• Contamination, saturation and Goodhart's law are the three standard reasons a benchmark score decays. Fresh, held-out items are the only reliable cure.
• A score supports a narrower claim than the headline. Write the narrow claim down.
• Formatting and parsing choices can shift scores by more than the difference between models. Ask how the number was produced.
• Public benchmarks are useful for tracking trends over years, not for deciding between two models that differ by two points.