Making AI Evaluations Safer: Implementing and Assessing a Simple Monitor That Reviews Actions Step by Step

1. Are inference calls that could potentially trigger security incidents sufficiently covered by monitoring? 1a. Do the criteria we use to define the scope of monitoring encompass the inference calls most likely to trigger security incidents? According to our criteria, agents in evaluations must be monitored if their capabilities meet or exceed Opus-4.7 or GPT-5.4 and the task meets either of the following conditions: Recent security incidents that went undetected for some time seem primarily due to related evaluations not being included in the monitoring scope. In section 5.5.1.1 of *Mythos Preview Alignment Risk Update*, Anthropic notes that during cybersecurity evaluation security incidents occurring between April and July (see section 5.5.1.3), internal monitoring did not cover most evaluations. The situation with OpenAI is similar: its chain-of-thought monitor covered most internal programming agent use cases and many reinforcement learning training runs, but it did not cover the cybersecurity evaluations that triggered the Hugging Face security incident. 1b. Are all inference calls meeting the criteria actually enabled for monitoring? 1b(i). For known inference calls meeting the criteria, are monitors enabled for all of them? 1b(ii). Have we accounted for nearly all inference calls without missing large-scale unknown call sources? 2. Among inference…