TASTE: Can AI Models Judge AI Safety Research Proposals?

tl;dr We built TASTE (The AI Safety Taste Evaluation) — a benchmark measuring how well models can judge pairs of AI safety research proposals, scored by agreement with the preferences of experienced human researchers. Two design choices were important for building a high-agreement benchmark (92 pairs, 77% estimated human agreement): a discussion stage in which researchers talk through disagreements before revising their scores, and filtering researchers’ labels for self-reported "strong" confidence. We find models perform worse than human researchers on TASTE (Fable 5, 60%). 📝Blog, 📄 Paper This work was done as part of the Anthropic Fellows Program. Background While some aspects of AI safety research are relatively straightforward to measure, progress on many questions in AI safety cannot be evaluated with verifiable rewards. For instance, research into mitigating risks from AI misalignment often involves forecasting risks posed by future AI systems. Another example is detecting when models are deceptive, which depends on the difficult task of accurately attributing beliefs and intentions to models. If we want to automate AI safety research — which might become necessary if automated AI research and development outpaces our ability to mitigate the risk of misalignment and misuse — we need reliable measurements of models’ capabilities on the hard-to-verify parts of safety research. One important hard-to-verify aspect of the research process is evaluating research proposals. Judging research proposals well is a high-leverage way to improve research quality, since some proposals are often much more important or tractable than others, and picking a poor initial direction can waste substantial time or resources. In order to evaluate models on the hardest-to-verify tasks, we have to use human judgment as ground truth. However, humans often disagree, making it unclear what the ground truth should be. This is why we focus on finding cases where humans agree and use agreement with humans as our main metric. Building a Research Judgment Benchmark (TASTE) In our paper, we present TASTE (The AI Safety Taste Evaluation), a benchmark of experienced AI safety researchers’ preferences over empirical safety research proposals. Our benchmark contains 92 pairwise comparisons, and we estimate human researcher agreement with our benchmark’s labels at 77%. In general, getting high agreement over fuzzy outputs, such as research proposals, is difficult. A few aspects of our benchmark design are important for improving the quality of human preference labels, measured by agreement with other researchers. We build the benchmark in three stages. First, we use a prompt scaffold with Claude Opus 4.6 to generate AI safety research proposals. Then, we recruit AI safety researchers to rate and comment on the research proposals, reporting their confidence for each set they score. Finally, we filter the dataset of human feedback to retain a high-agreement set of preference pairs. Figure 1: TASTE construction pipeline. Researchers scored each proposal 1–5 on three axes (overall, high-level, approach), ranked the three proposals with ties allowed, and reported a per-prompt confidence. Benchmark preferences are derived from the overall scores. To create the model-written research proposals, we started with a collection of 93 human-written research proposals presented in Anthropic’s Fellows Program. We used Claude Opus 4.6 to reverse-engineer prompts that could motivate each human proposal, and supplemented this set with prompts we wrote ourselves. We generated research proposals using a prompt scaffold that varied paper summaries in context for diversity. See the paper for the full proposal and more details on the scaffold. Figure 2: Example motivating-question prompt and extract from a model-generated research proposal (full proposal in the paper). To construct a human feedback dataset using those research proposals, we asked AI safety researchers to evaluate groups of proposals generated for the same motivating-question prompt. The researchers first gave individual feedback, then discussed disagreements in pairs, and then revised their preferences. Figure 3: Pair-discussion protocol. For each prompt, four researchers gave their feedback individually, discussed disagreements in pairs, then revised their feedback. See the paper for the rubric shown to the researchers. We find that this discussion stage, in combination with filtering for self-reported “strong” confidence, raises estimated human agreement by 15 percentage points, from 53% pre-discussion to 68% for strong-confidence, post-discussion preferences. By looking at discussion transcripts, we find multiple reasons for disagreements among human researchers: substantive disagreements over how well particular techniques would work, and more mundane causes such as one person misreading a proposal text the first time around. We give a fuller breakdown of reasons in the paper. Figure 4: Str