Building a Golden Eval Dataset from Production Traffic

You update a prompt, or a provider rolls out a new checkpoint under the same model ID, and something in production regresses. The last week of user complaints looks slightly different from the week before. A public benchmark like MMLU won’t catch that. It measures general capability across academic subjects, not how the model handles your product’s traffic. A golden eval dataset closes that gap. It’s a curated collection of production inputs paired with reviewed expected outputs, versioned in Git, and run before every deploy. It answers the question a benchmark can’t. Does this change help or hurt on the traffic you serve? This guide covers what a golden set is, why production data is a better foundation than synthetic data, and a five-step process for building one from live traffic. It also covers how to run the same set against many candidate models through one API, so you can pick your next model on evidence from your own traffic rather than leaderboard rank. A golden eval dataset is a curated set of production inputs paired with reviewed expected outputs, used as a regression test before every meaningful change. Once the set exists, you can run it across many models…