aiqbench takeAI Safety

The author argues that when AI monitors (systems that review other AI agents' actions for danger) fail, the cause is often misalignment in the monitor itself rather than insufficient capability. Monitor evaluations face the same difficulties as evaluations of agent alignment, so alignment evals alone are weak evidence that a monitor can be trusted. Readers working on agent oversight should treat monitor assessments as a separate, multi-input problem.

Sources coveringLessWrong, Google Cloud Blog, Quantum Zeitgeist, Latent Space

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Control monitors need alignment assessments too

LessWrongOctober 24, 202611 min read

TL;DR: When misalignment monitors fail, it is already often because of alignment failures, not because they are too weak. I expect this to continue in the future. Monitor evals share similar difficulties with agent alignment evals - which is bad news since getting evidence about agent alignment using alignment evals is hard. Alignment eval results should not be the only input to monitor alignment assessments, similar to how alignment evals should not be the only input to agent alignment assessments. These are my personal views; they do not represent the views of my employers. Some of the worst past monitor failures are mostly alignment issues Monitors are sometimes misaligned in non-scheming ways that have caused some important monitor false negatives: They sometimes fail to balance different priorities the way developers would like them to For example, they may fail to balance broad safety and broad ethics: see e.g. 5.2.2 Partial refusals on safety work undermining stress-testing research Monitors might also not have the right information about the state of the system, what invariants should be maintained, or what is most important to surface. So these are not pure alignment failures. They sometimes inherit (and may re-derive) the motivated beliefs that caused an agent to take dangerous actions e.g. they missed the propensity of Mythos 5 to do supply chain attacks in pen-testing evals In the future, the most dangerous monitor failure modes will likely also be alignment issues. For example, monitors might collude with the agent (i.e. they might pursue misaligned goals by strategically not raising dangerous agent actions for review if they expect that not raising such an action would empower agents with values they expect to be aligned with their own). While collusion would be the most damaging kind of monitor alignment failure if it happened, I think previous work often missed that this is not the only way in which failures of monitor alignment could be catastrophic. Monitor quality is not just a capability question Monitor red-teaming often focuses on estimating monitor capabilities (or noticing whether developers are not telling monitors some crucial information[1]), and the current arguments in risk reports about monitors are also mostly aimed at monitor capabilities. Previous work and arguments did study the possibility of monitors being schemers, ruling it out by arguing they are too weak to be scheming, and concluding such monitors are trusted. But Anthropic's Opus 4.8 misalignment monitor is very unlikely to be scheming, and Opus 4.8 is probably smart enough to understand and catch the false negatives above. So more of the same kind of red-teaming would probably not have helped catch Opus 4.8's misalignment-as-a-monitor. Therefore we need to assess the monitor’s alignment. Where monitor evals fail Getting evidence about agent alignment is hard in part due to rare agent failures being potentially catastrophic. This is less of an issue for monitor evals since monitors “just” need to catch most things for control to be effective. But getting evidence that monitors are aligned enough can still be hard, because the situations where the agent does sth bad are way more likely to be the ones where the monitor is also wrong. And thus monitor evals can suffer from two other core difficulties agent alignment evals face: The search space is big: In the same way automated behavioral evaluations missed the extent to which Mythos 5 could be reckless and show biased reasoning, classic monitor evals may have also missed Opus 4.8's failure to flag such behavior as misaligned. Monitors can be eval aware too: Mitigating eval awareness may be as hard as when trying to catch AIs with misaligned goals in automated behavioral evaluations. And it may get worse if AI companies train harder against monitor evaluations, in the same way training against honeypots likely makes eval awareness worse in agents. Alignment assessments are much less important when evaluating misuse monitors, where the alignment of the user is basically not correlated at all with the alignment of monitors, such that monitor red-teaming is basically in-distribution for things the monitors should defend against. Alignment evals are not the only way we assess agent alignment ⇒ evals should not be the only way we assess monitor alignment The arguments from the Anthropic risk report about agent alignment can usually be converted into arguments about the alignment of monitors: Agent alignment arguments Potential monitor alignment arguments Claim: agents ~never do malign stuff Claim: if agents do malign stuff, monitors would likely catch them Argument 1: Lack of strong covert capabilities Agents don’t have strong enough covert abilities to tamper with our evidence of alignment? Monitors don’t have strong enough covert abilities to tamper with our evidence of alignment? Argument 2: Experience with prior models We are not aware of misalig

When misalignment monitors fail, it is already often because of alignment failures, not because they are too weak. I expect this to continue in the future.

Monitor evals share similar difficulties with agent alignment evals - which is bad news since getting evidence about agent alignment using alignment evals is hard.

Alignment eval results should not be the only input to monitor alignment assessments, similar to how alignment evals should not be the only input to agent alignment assessments.

These are my personal views; they do not represent the views of my employers.

They sometimes fail to balance different priorities the way developers would like them toFor example, they may fail to balance broad safety and broad ethics: see e.g. 5.2.2 Partial refusals on safety work undermining stress-testing researchMonitors might also not have the right information about the state of the system, what invariants should be maintained, or what is most important to surface. So these are not pure alignment failures.

For example, they may fail to balance broad safety and broad ethics: see e.g. 5.2.2 Partial refusals on safety work undermining stress-testing research

Monitors might also not have the right information about the state of the system, what invariants should be maintained, or what is most important to surface. So these are not pure alignment failures.

They sometimes inherit (and may re-derive) the motivated beliefs that caused an agent to take dangerous actions e.g. they missed the propensity of Mythos 5 to do supply chain attacks in pen-testing evals

e.g. they missed the propensity of Mythos 5 to do supply chain attacks in pen-testing evals

In the future, the most dangerous monitor failure modes will likely also be alignment issues. For example, monitors might collude with the agent (i.e. they might pursue misaligned goals by strategically not raising dangerous agent actions for review if they expect that not raising such an action would empower agents with values they expect to be aligned with their own).

While collusion would be the most damaging kind of monitor alignment failure if it happened, I think previous work often missed that this is not the only way in which failures of monitor alignment could be catastrophic.

Monitor red-teaming often focuses on estimating monitor capabilities (or noticing whether developers are not telling monitors some crucial information[1]), and the current arguments in risk reports about monitors are also mostly aimed at monitor capabilities.

Previous work and arguments did study the possibility of monitors being schemers, ruling it out by arguing they are too weak to be scheming, and concluding such monitors are trusted.

But Anthropic's Opus 4.8 misalignment monitor is very unlikely to be scheming, and Opus 4.8 is probably smart enough to understand and catch the false negatives above. So more of the same kind of red-teaming would probably not have helped catch Opus 4.8's misalignment-as-a-monitor.

Therefore we need to assess the monitor’s alignment.

Getting evidence about agent alignment is hard in part due to rare agent failures being potentially catastrophic. This is less of an issue for monitor evals since monitors “just” need to catch most things for control to be effective.

But getting evidence that monitors are aligned enough can still be hard, because the situations where the agent does sth bad are way more likely to be the ones where the monitor is also wrong. And thus monitor evals can suffer from two other core difficulties agent alignment evals face:

The search space is big: In the same way automated behavioral evaluations missed the extent to which Mythos 5 could be reckless and show biased reasoning, classic monitor evals may have also missed Opus 4.8's failure to flag such behavior as misaligned.

Monitors can be eval aware too: Mitigating eval awareness may be as hard as when trying to catch AIs with misaligned goals in automated behavioral evaluations. And it may get worse if AI companies train harder against monitor evaluations, in the same way training against honeypots likely makes eval awareness worse in agents.

Alignment assessments are much less important when evaluating misuse monitors, where the alignment of the user is basically not correlated at all with the alignment of monitors, such that monitor red-teaming is basically in-distribution for things the monitors should defend against.

Agents don’t have strong enough covert abilities to tamper with our evidence of alignment?

Monitors don’t have strong enough covert abilities to tamper with our evidence of alignment?

We are not aware of misaligned agent behavior?

We are not aware of misaligned monitor behavior?

Our agent monitoring is good?

Our monitoring of monitors is good?

Agents are similar to prior agents?

Monitors are similar to prior monitors?

We tried building some model organisms of agent misalignment?

We tried building some model organisms of monitor misalignment?

We didn’t see agents do much misaligned stuff?

We didn’t see monitors do much misaligned stuff?

And we would have caught misaligned agent behavior?

And we would have caught misaligned monitor behavior?

Training does not encourage misaligned agent behavior directly (per training monitoring)?

Training does not encourage misaligned monitor behavior directly (per training monitoring)?

Indirect pressures seem unlikely?

Agent alignment evals didn’t find scary stuff?

Monitor alignment evals didn’t find scary stuff?

Eval awareness is not too bad / we mitigated it?

We didn’t overfit to agent alignment evals?

We didn’t overfit to monitor evals?

We caught agent misalignment model organisms?

We caught monitor misalignment model organisms?

But considerations around agent-monitor correlations make monitor alignment assessments different from agent alignment assessments.

Try to understand how decorrelation works (and whether this would have helped with various model organisms and past incidents)Does using models from different labs help?Does sanitizing the transcripts in various ways help?Do certain kinds of fine-tuning help?Do more unsupervised summarization and clustering have the same issues as classification?

Does using models from different labs help?

Does sanitizing the transcripts in various ways help?

Do certain kinds of fine-tuning help?

Do more unsupervised summarization and clustering have the same issues as classification?

Try harder at automated-behavioral-eval-for-monitors (for example using approaches like Screwtape). Maybe try steering interventions against eval awareness.

^For monitors, lack of information can be catastrophic. For agents it’s usually not the case: you need to be quite reckless to take actions that could plausibly be catastrophic without checking you are not doing something catastrophic! Agentic monitors may solve this information issue and just turn it into a regular capability+alignment issue.

For monitors, lack of information can be catastrophic. For agents it’s usually not the case: you need to be quite reckless to take actions that could plausibly be catastrophic without checking you are not doing something catastrophic! Agentic monitors may solve this information issue and just turn it into a regular capability+alignment issue.

Full text under the publisher's open licence · 1335 words

Read at LessWrong ↗Ask about this article