Tracing Eval-Awareness Emergence Through Training of OLMo 3

TL;DR Recent work from Goodfire & UK AISI – Verbalized Eval Awareness Inflates Measured Safety – shows that newer open-weight models verbalize evaluation-awareness (VEA) more often, and that this inflates measured safety. Between OLMo-3-32B-Think and OLMo-3.1-32B-Think – identical base, SFT, DPO, and RL data, differing only in an additional ~3 weeks of the RLVR stage – VEA roughly doubles. Because OLMo ships stepwise checkpoints across all training stages, we can attribute VEA growth to specific points in the pipeline. Measuring VEA across pretraining, the SFT→DPO→RLVR stages at various points on five safety benchmarks, we find: VEA is essentially negligible during pretraining (~1%). It is increased substantially by SFT, collapsed by DPO, and increased again by RLVR. The increase in SFT is likely driven by the SFT data containing VEA, particularly on safety prompts. Eval-gaming behaviour (difference in refusals with or without VEA) roughly increases throughout RLVR (but with high variance). Given OLMo is quite different from how current frontier models are trained, it is unclear whether this analysis would generalise to those models. However, as a somewhat natural setting in which to study evaluation awareness and eval-gaming emergence we think it could still produce interesting insights. We think investigating how and why RLVR in particular increases VEA and eval-gaming is the exciting next step. These results are also relevant for efforts to produce model organisms of evaluation awareness and evaluation gaming. The fact that SFT increases VEA by training on VEA makes that finding less interesting, but the RLVR behaviour suggests eval-aware MOs may benefit from similar training if the aim is to induce more natural eval-gamin behaviour. Motivation Current studies of eval-awareness don't pin down which parts of post-training drive its increase. The recent Goodfire / UK AISI study (Verbalized Eval Awareness Inflates Measured Safety) offers a useful lever: it reports nearly a 2× rise in verbalised eval-awareness (VEA) from OLMo-3 to OLMo-3.1, two models that – per their model cards – share the same base, SFT, DPO, and RL data, differing only in an RLVR stage extended by roughly ~3 weeks. This shared lineage lets us study eval-awareness emergence as a function of training step and stage. The study measures VEA by sampling models on safety and red-team benchmarks (Fortress, JailbreakBench, AdvBench, HarmfulQA, and AgentHarm, among ~19 total) and using an LLM judge to flag whether a model's chain-of-thought explicitly considers that it might be under evaluation. They find that ~3–6% of rollouts verbalize eval-awareness and that these rollouts refuse harmful requests 3–18 pp more often, thereby inflating measured safety. We adopt their exact LLM judge and the same safety benchmarks. Method (more details in Appendix): VEA is judged with the same rubric as mentioned in the Appendix section of Goodfire / UK AISI work on Verbalised Eval Awareness inflating safety, applied to the model's chain-of-thought (the text before </think>); a rollout counts as VEA if the judge returns a non-None quote. All runs use the same 444 safety prompts, 8 rollouts each. VEA is negligible during pretraining We sampled five pretraining checkpoints of the base model (steps 0 to 656,000) and judged VEA in the chain-of-thought. Because early checkpoints often fail to even follow the CoT format, we report two denominators: VEA among rollouts that produced a </think> (coherent CoTs), and VEA over all rollouts. Every rollout throughout this work uses the same fixed set of 444 safety prompts, drawn from the benchmarks used in the Goodfire/UK AISI study: 100 each from Fortress, JailbreakBench, AdvBench, and HarmfulQA, plus 44 from AgentHarm. These are adversarial / red-team prompts (harmful requests, jailbreak attempts, and agentic-misuse scenarios). Figure 1. Even at the final pretraining checkpoint, only ~1.2% of chains-of-thought verbalise eval-awareness. What grows is the ability to follow the CoT format (the annotated </think> rate climbs 0% → 68%), not the tendency to flag "this is a test." The dashed line marks the released base main model. RLVR amplifies VEA We note that the VEA increase between OLMO 3 and 3.1 must be due to the increased RLVR. So, we start by measuring the trend of VEA on OLMo-3.1-32B-Think at ten RLVR checkpoints (steps 50 to 2300) across all five benchmarks. Figure 2. VEA across five benchmarks through RLVR VEA increases by roughly 2–2.5× over the RLVR sweep on Fortress, and ~1.5× on average across all benchmarks (start to end). Absolute rates vary strongly by benchmark: prompt-only harmful-compliance benchmarks elicit much higher VEA than Fortress, whose more realistic, context-rich prompts elicit it least often. Although the increase is not strictly monotonic, the local ups and downs are broadly synchronised across benchmarks, suggesting a shared propensity adjusted by benchmark-specific adjustments to how fr