Held-out Monitors Sometimes Degrade, Even When Not Trained Against

Thanks to Dennis Akar, Rauno Arike, Shubhorup Biswas, Claude Fable, Max Heitmann, Vladimir Ivanov, and Jordan Taylor (alphabetical order) for helpful comments on this draft and on the research so far. Thanks to Rohan Subramani and Rhys Ward for high-level comments and discussion. Based on project proposals from Max Heitmann, Jordan Taylor, and Joshua Clymer. This work was done while at Aether Research.

Code available here, metrics & run info available here.

TL;DR: Held-out evals / monitors / probes would be really nice to have, but the "held-out-ness" is easier claimed than guaranteed. We measure a generalized form of "feedback spillover" and show that training against an LLM monitor can sometimes degrade a deception probe, and vice versa.

It seems crucial to have measures of alignment that still work, even though we train on other measures of alignment. Whether we get this by default is an open question.

We run preliminary experiments on a suite of probes and LLM monitors, and report the following: Training against one proxy can produce reward hacking policies that are less suspicious. Proxies also become worse at discriminating hacks from non-hacks, even when not trained against. We can observe the correlated degradation of different proxies, and note that on the occasions when a strong monitor is evaded, other monitors are evaded even more strongly.

Training against one proxy can produce reward hacking policies that are less suspicious.

Proxies also become worse at discriminating hacks from non-hacks, even when not trained against.

We can observe the correlated degradation of different proxies, and note that on the occasions when a strong monitor is evaded, other monitors are evaded even more strongly.

Our preliminary recommendation: if you are using a held-out proxy to evaluate your model, you should have empirical or theoretical reasons to assume it won't degrade during training.

A science of held-out monitors seems possible; we currently don't have one. This makes safety cases that rely on held-out alignment proxies less reassuring than hoped.

It seems like the default "alignment plan" is going to be haphazard, relying on defense-in-depth to make up for a lack of fundamental breakthroughs. In these worlds, we'll often have to train on proxies for the thing we want, or against proxies of misalignment, while leaving some held-out evals to validate our methods[1]. If that's the case, we want to make sure our monitors / evals are uncorrelated—especially in the sense of "training on proxy X doesn't degrade the ability of proxy Y to keep monitoring the model".[2] Specifically, it would be really bad if labs thought they had some held-out evals, but it turns out that their training tended to degrade the usefulness of the held-out evals significantly.

Here, we measure this explicitly, and show tentative evidence that this cross-degradation indeed happens. While it was previously known that LLM monitors can be fooled, and probes can degrade due to representational drift, we show that in some circumstances, when one degrades, so does the other—that is, "feedback" from one proxy can "spill over" to degrade the other as well. We also find that sometimes a proxy can be evaded without being degraded — that it can still separate aligned behavior from misaligned behavior, but the misaligned behavior looks less and less suspicious anyways. This isn't neat, and it doesn't happen all the time, but we have some evidence that this generalized feedback spillover does sometimes occur. We hope that we (and others) can build on this to make a real science of held-out monitors.

We want to evaluate the degradation of held-out evals, probes, or monitors (which we'll bundle together and call proxies, for proxies of alignment). To that end, we train Qwen3-8B in a difficult coding environment where simple reward hacking (i.e. hardcoding) is incentivized. We then trained against an LLM monitor or a probe,[3] and evaluated changes in hack/non-hack separability, AUROC, and mean suspiciousness of both the trained-against proxy and the held-out proxies. We then evaluated (1) whether the proxies degraded, (2) if so, how, and (3) whether the policy that learned to evade the train-against proxy would now be capable of evading the held-out proxies as well.

Much of our setup comes from The Obfuscation Atlas. We train on MBPP-Honeypot, a one-turn coding environment with a very easy reward hack (the model is only scored based on whether its produced code satisfies a public test, so it is easily hardcodeable).

As in the Obfuscation Atlas, we RL[4] on this environment, giving the model reward equal to , where Rtask is 1 if the test was passed, and 0 otherwise. Pmonitor is the proxy's normalized suspiciousness score — for LLM monitors, this is obtained by showing an LLM all or part of the policy's outputs and asking for a "suspiciousness score", from 0-10 (which we then normalize to 0.0-1.0). For a probe, we use the Obfuscation Atlas' methodology again to get Pmonitor by looking at the final probe score after asking a follow up question about whether or not the model was reward hacking. λ is our monitor penalty — many of the runs we report on use λ = 1.0, but as we were trying to encourage reward hacking, we also tried λ = 0.5, and a schedule of increasing λ linearly from 0.0 to 1.0 during the run. For more info about training, look at the configs.

We used two classes of proxies in this set of experiments: LLM Monitors and Linear Probes.

We selected our two LLM monitors by having a large suite of LLMs attempt to classify the example hacks in the MBPP-Honeypot dataset, and taking two monitors with high, although not perfect, ROC-AUCs.[5] Haiku 3 was the weaker of the two (ROC-AUC 0.9), while Deepseek-chat was better at distinguishing hacks from non-hacks (ROC-AUC 0.99). We tested output-only and CoT+output variations of both monitors.

We trained probes using the methodology from The Obfuscation Atlas — we train two probes on the initial model, and we get probe scores by running the policy model's text through the frozen model[6] and using these activations. (Note that this means that any probe degradation is due to a change in the policy model's behavior, rather than representation drift or obfuscated activations. That is, a probe getting worse means the policy model's text has changed, rather than it simply learning to store "deception" along a different direction than the probe checks.) Our two probes are:

Probe 2: In-distribution "reward hacking on MBPP probe", trained on the off-policy, human-written honest solutions and reward hacks to these particular problems.

Since the policy model is penalized for the suspiciousness score the monitor gives it, and the monitors start out by being pretty sure the clean examples are clean, we expect the policy model to achieve a high reward by learning to hack in unsuspicious ways.[7] We measure this by looking at μhack, the mean suspiciousness score of all the hacking transcripts, and how this changes over the course of a run where the model learns to hack despite the monitor[8].

[Fig 2] When we train against a monitor (given as the row label), how much less suspicious do the hacks become? This is the drop in mean suspiciousness (μhack) to each monitor over the course of the run, with 5-95% CIs. (Note this chart is absolute rather than percentage-based, so 0.5 means that, if the model s