Improving Audit Realism With Inference Time Compute and Deployment Scaffolds

tl;dr:  We introduce critique refinement, a method to iteratively improve the realism of auditor outputs in Petri, and DISH (Deployment Imitating SWE-agent Harness), a method to audit a model within a coding agent scaffold to ensure the model receives a real system prompt, tools and scaffold injections. These techniques compose to triple realism win rate and substantially reduce unprompted verbalized eval awareness in Petri. Scaling critique refinement leads to additional improvements in realism and eval awareness. Our full paper is released here. This work was carried out as a collaboration between Meridian Cambridge, Cambridge Boston Alignment Initiative, UK AI Security Institute and Anthropic. Introduction Automated alignment auditing tools like Petri are an important component of safety assessments, including the alignment evaluations in recent frontier model system cards (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card). These cards also flag evaluation awareness as a challenge to audit validity: models can distinguish evaluations from deployment data, and their reasoning indicates this may affect their subsequent actions. Reducing evaluation awareness increases misaligned behavior in Claude models (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card), yet lower evaluation awareness is associated with less concerning behavior in other models (Rajamanoharan et al., 2026; Lindner et al., 2026). Petri relies on LLMs to simulate system prompts, user messages, tools and tool call responses when auditing another model. This simulation is often imperfect, reducing evaluation realism and increasing evaluation awareness. Previous attempts to improve simulation quality have used deployment resources (Kissane et al., 2026), resampled deployment trajectories (Williams et al., 2026; Ivanov et al., 2026) and allowed models to interact with a real environment instead of simulating all tool call results (Claude Fable 5 & Mythos 5 System Card). These methods improve realism, but are a one-off increase, and require gathering relevant resources and deployment data for each task (or set of tasks). Instead, we tackle the problem by introducing critique refinement, a protocol to iteratively improve the realism of auditor actions. This means improvements can scale with inference-time compute in a general way. We also introduce DISH, which places the target model in a real coding agent scaffold to ensure the model is audited with a real system prompt, tools, and scaffold injections. Both methods are general: critique refinement can work with any Petri seed and DISH can work with any Petri seed evaluating coding agent behavior.  Method Critique Refinement Figure 1: Critique refinement improves the auditor's actions iteratively with feedback and chooses the best next action with a champion-challenger protocol for depth scaling and a single elimination tournament for breadth scaling. Our first technique is critique refinement, a protocol to iteratively improve the realism of auditor actions. Following the generate-feedback-refine pattern (Madaan et al., 2023), the auditor generates a candidate action and presents it to the target model. The target model generates feedback on the realism of the candidate action. We also use the target model to select the most realistic candidate action so far using a champion-challenger system: we present the target model with the previous round’s winner for most realistic action and the new candidate action, and ask it to select the more realistic action. There are two ways to scale critique refinement: breadth and depth. To scale depth, we run more rounds of champion-challenger to allow the auditor more turns to iteratively refine an action. To scale breadth, we run multiple rounds of champion-challenger in parallel, and then place the champions in a single-elimination tournament where the target model selects the more realistic action given two choices. cr2bo4 denotes a depth of 2 (cr - critique refinement) and a breadth of 4 (bo - best of), and cr4 denotes a depth of 4 and a breadth of 1 (bo1 left out for brevity). DISH Figure 2: DISH. We place the target model in an agent harness to supply it with a real system prompt, tools, and scaffold injections like system reminders. The auditor model interacts with the target model through the agent harness. We introduce DISH to further improve the realism of Petri audits in the coding agent setting. DISH places the target model in a real coding agent scaffold: the scaffold provides the system prompt, tools, and scaffold injections such as system reminders, and the auditor controls user messages and tool call responses. DISH builds on the success of using realistic system prompts and tools in Petri (Kissane et al., 2026; Souly et al., 2025). Prior to these methods, the Petri auditor needed to simulate all scaffold prompts from scratch. Currently Claude Code, Codex CLI, and Gemini CLI are supported. DISH is released a