Logits as a new monitor for evaluation awareness

TL;DR: We build a logit monitor for eval awareness: throughout the CoT, we estimate an LLM's probability of producing an eval-aware sentence. The logit monitor outperforms LLM judge monitoring of verbalized eval awareness, using 10× to 100× fewer rollouts, on Kimi K2.5 and Qwen 3 32B, across two tasks: separating evaluation prompts (Fortress & Petri) from deployment prompts (WildChat) predicting which evaluation prompts will elicit rollouts with verbalized eval awareness. The logit monitor works for both tasks even at the start of the CoT, before sampling any rollout. Also, the monitor is largely unaffected by whether a rollout verbalizes eval awareness. This suggests that the probability that a rollout contains verbalized evaluation awareness is primarily set by the prompt rather than emerging over the chain of thought. Author: Santiago Aranguri (Goodfire) Introduction Language models can recognize they are being tested, but how do we know when they do? Evaluation awareness — a model’s ability to distinguish between evaluation and deployment contexts — can make evaluations unreliable in ways that are hard to detect. Sometimes models verbalize this recognition, but this may be present internally without appearing in the output. [1] [2] Instead of waiting for a model to verbalize eval awareness, we measure how likely it is to do so. We build a logit monitor that reads the model's output logits to estimate the probability it would produce a sentence expressing eval awareness. Our logit monitor is complementary to activation probes, which are based on the internals of the model, and sits in a middle ground: a richer signal than verbalizations, and one that may be easier to interpret than activation probes. The logit monitor outperforms monitoring verbalized eval awareness, while using 10× to 100× fewer rollouts for Kimi K2.5 and Qwen 3 32B. We show this on two tasks: separating evaluation prompts (Fortress, Petri) from deployment prompts (WildChat), and predicting whether an evaluation prompt will produce at least one rollout displaying verbalized eval awareness. The logit monitor already works for both tasks even if we only read the logits at the start of the CoT. The logit monitor’s value is largely unaffected by whether a rollout verbalizes eval awareness. Logits are a cheap and interpretable resource to monitor eval awareness, complementary to verbalizations and activations. Methods The logit monitor calculates the probability the model would produce a prespecified eval-aware sentence (such as "The user might be testing") at every sentence boundary in the chain of thought. The logit monitor doesn’t require training: we token-force the eval-aware sentence, read the probability the model assigns to each of the tokens in the sentence, and multiply these to get the exact probability it would produce the sentence (Appendix A).[3] The verbalization monitor samples k rollouts and scores a prompt by the fraction of rollouts that verbalize eval awareness, as judged by an LLM. We use Fortress and Petri as evaluation prompts and WildChat as deployment prompts. Results We first give a qualitative picture of our results: the logit monitor is higher on evaluation than deployment prompts, is largely unaffected by whether the rollout contains verbalized eval awareness, and shows signal from the start of the CoT (Figure 1). On an individual Fortress prompt, rollouts with and without verbalized eval awareness average to similar curves (Figure 1, left). Averaging across Fortress prompts that have at least one rollout with verbalized eval awareness out of 100, the two types of rollouts still trace similar curves. These prompts also score higher than Fortress prompts that produce no rollouts with verbalized eval awareness, which in turn score higher than WildChat prompts (Figure 1, right). Figure 1. The logit monitor is elevated on evaluation prompts and shows signal from the start of the CoT. For each rollout, we compute the probability of "The user might be testing" at every sentence boundary in the CoT. Left: logit monitor averaged across rollouts from a single Fortress prompt, split by whether the rollout verbalizes eval awareness. Right: logit monitor averaged across rollouts and prompts. Red and orange: rollouts from Fortress prompts with at least one rollout displaying verbalized eval awareness out of 100, split by whether the individual rollout verbalizes it. Blue: rollouts from Fortress prompts with no rollout displaying verbalized eval awareness. Green: rollouts from WildChat prompts. Shaded regions show 95% bootstrap CIs. We now provide our key results: The logit monitor outperforms the verbalization monitor on two tasks, while using fewer rollouts (Figure 2). For each prompt with k rollouts, the logit monitor’s score is the maximum of the logit monitor across rollouts and CoT positions. The verbalization monitor’s score is the fraction of those rollouts showing verbalized eval awareness, as judged by an LLM.