FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth

Abstract:Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies dense, executable ground truth. Each task presents eight ingredients and asks for a three-ingredient portfolio; before model execution, Epicure scores all 56 possible portfolios. We evaluate 27 frontier endpoints on an identical 534-task core spanning substitution, pairing, and constrained composition. Every ranked model has exactly 89 valid responses per panel and family (14,418 model-task cells total), eliminating differential missingness from the leaderboard. The FlavourBench Score is the equal-family mean of the frozen task scores. We use 50,000 anchor-cluster bootstrap replicates for simultaneous 95% score bands and 100,000 sign-flip draws for all 351 paired model contrasts, with Holm control. The two independently compiled panels correlate at r = 0.89 (rank rho = 0.80). Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0-69.2); 101 of 351 model pairs are resolved. The release includes the prompts, all portfolio score maps, raw responses, exact routes, content hashes, and an offline verifier that reconstructs every result.

10 pages, 5 figures. Evaluation of 27 frontier language-model endpoints on 534 identical tasks per model, comprising 14,418 scored model-task cells. Code: this https URL Dataset: this https URL Interactive leaderboard: this https URL

Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG); Software Engineering (cs.SE)

arXiv:2608.20574 [cs.AI]

(or arXiv:2608.20574v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2608.20574 arXiv-issued DOI via DataCite (pending registration)