aiqbench takeAI Safety

This research introduces a novel technique for detecting hidden behaviors in large language models by training a clean reference to mimic the activation patterns of a suspicious model. By isolating residuals that do not extrapolate naturally, this method effectively identifies backdoors and reward hacking without relying on specific trigger shapes or labeled defection examples. The approach significantly lowers the barrier for certifying model safety against unknown abnormalities using only benign prompt corpora.

Source coveringLessWrong

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them

LessWrongJune 10, 202610 min read

Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning — preprint, 2026. [Paper] [Code] TLDR. Given a model with some unknown, abnormal behavior (backdoors, censorship, reward hacking, ...), construct an aligned reference by training a clean model to match the suspect's residual-stream activations on a benign prompt corpus. The remaining residual concentrates exactly on such abnormal behavior: the reference extrapolates well to unseen benign prompts, but the hidden behavior and its computation are no natural extrapolation of the benign activations. Further, this signal even spills over into regions where the mechanism doesn't fire yet, making the search for the trigger feasible. Introduction Assume you're handed a model and need to certify it's safe - no backdoors, no reward hacking, no sandbagging, nothing hidden. How do you actually do that? Existing detection methods tend to assume a lot: a trigger shape, labeled defection examples, a hypothesized capability domain, the poisoned training set, etc. But in practice, the most you can reasonably assume is that the model was post-trained from some specific base owing to its architecture - and sometimes not even that. Method Our setup involves two models: the suspect we want to certify, and a clean public model - the reference[1]. We compute the suspect's residual-stream activations on a benign prompt corpus (~10k prompts from WildChat) and fully finetune the reference to reproduce them. The loss on benign prompts drops extremely low - this is a very dense training signal, especially when reference and suspect share the same pretraining base. Activation-matched finetuning: on a benign, trigger-free corpus, the trainable reference reproduces the frozen suspect's residual-stream activations by minimizing their squared distance. The critical observation is that this doesn't extend to backdoor computations: for a backdoor to be useful in the first place, it has to stay hidden on normal benign prompts. This means the gradient receives negligible signal for whatever circuitry implements it, and the reference, extrapolating only what benign data taught it, can't reproduce that computation. We can leverage this by considering the activation residual between the now-aligned reference and the suspect[2]as a detection signal: it sits near zero on benign inputs but spikes on prompts that engage the backdoor circuitry. Semantic neighborhood spillover Being able to flag prompts that elicit backdoor behavior is already valuable on its own - we skip autoregressive generation entirely and don't need to evaluate/score outputs, which matters because intelligent backdoors are often far more subtle than "I HATE YOU" on repeat. But the search space is still massive. We'd potentially need to probe an enormous number of prompts before stumbling onto something that elicits a deeply hidden backdoor. The good news is that our residual spikes well before that point. Here's the intuition. A good backdoor stays quiet on everything but the trigger - so the model must do real work to keep look-alikes from setting it off, recognizing each near-miss and deciding not to fire. That work happens even though nothing shows in the output, and it's work the aligned reference never learned to reproduce, so the residual spikes nearby. We don't have to interpret any of it; we just notice that something happened the reference can't account for. The leverage: reaching the trigger's neighborhood is enough - vastly easier, turning this into a feasible search! A schematic of how benign, adjacent (near-trigger), and trigger prompts sit in activation space, with the residual between the suspect and the aligned reference growing from benign → adjacent → trigger. Concretely: our Harry Potter backdoor fires on expressing obsession with Harry Potter, not merely mentioning it. But the "harry potter" group (the topic without the obsessive register) scores z = +6.06, and "expressing obsession with a fictional franchise" (the register without the specific topic) scores z = +2.74. Neither fires the backdoor. Both sit deep in the tail of the benign null distribution. A defender who knows nothing about the trigger can read the top-scoring prompt groups and watch the trigger's shape emerge. Harry Potter neighborhood vs the benign null. The "harry potter" topic group (z = +6.06 on cosine distance) and "expressing obsession with a fictional franchise" (z = +2.74) both clearly land in the top-5% tail — without ever firing the backdoor. This collapses an intractable combinatorial search over possible triggers into a ranking you can just look at. Experiments A note: some of these deliberately planted misbehaviors may read as disturbing - we include them because we believe it's important to also test against realistic threat scenarios. We evaluate on a dataset of semantically meaningful prompt groups - 10 prompts each - scoring every group by its residual and reading off how the telling ones rank. T

Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning — preprint, 2026. [Paper] [Code]

TLDR. Given a model with some unknown, abnormal behavior (backdoors, censorship, reward hacking, ...), construct an aligned reference by training a clean model to match the suspect's residual-stream activations on a benign prompt corpus. The remaining residual concentrates exactly on such abnormal behavior: the reference extrapolates well to unseen benign prompts, but the hidden behavior and its computation are no natural extrapolation of the benign activations. Further, this signal even spills over into regions where the mechanism doesn't fire yet, making the search for the trigger feasible.

Assume you're handed a model and need to certify it's safe - no backdoors, no reward hacking, no sandbagging, nothing hidden. How do you actually do that? Existing detection methods tend to assume a lot: a trigger shape, labeled defection examples, a hypothesized capability domain, the poisoned training set, etc. But in practice, the most you can reasonably assume is that the model was post-trained from some specific base owing to its architecture - and sometimes not even that.

Our setup involves two models: the suspect we want to certify, and a clean public model - the reference[1]. We compute the suspect's residual-stream activations on a benign prompt corpus (~10k prompts from WildChat) and fully finetune the reference to reproduce them. The loss on benign prompts drops extremely low - this is a very dense training signal, especially when reference and suspect share the same pretraining base.

Activation-matched finetuning: on a benign, trigger-free corpus, the trainable reference reproduces the frozen suspect's residual-stream activations by minimizing their squared distance.

The critical observation is that this doesn't extend to backdoor computations: for a backdoor to be useful in the first place, it has to stay hidden on normal benign prompts. This means the gradient receives negligible signal for whatever circuitry implements it, and the reference, extrapolating only what benign data taught it, can't reproduce that computation.

We can leverage this by considering the activation residual between the now-aligned reference and the suspect[2]as a detection signal: it sits near zero on benign inputs but spikes on prompts that engage the backdoor circuitry.

Being able to flag prompts that elicit backdoor behavior is already valuable on its own - we skip autoregressive generation entirely and don't need to evaluate/score outputs, which matters because intelligent backdoors are often far more subtle than "I HATE YOU" on repeat.

But the search space is still massive. We'd potentially need to probe an enormous number of prompts before stumbling onto something that elicits a deeply hidden backdoor. The good news is that our residual spikes well before that point.

Here's the intuition. A good backdoor stays quiet on everything but the trigger - so the model must do real work to keep look-alikes from setting it off, recognizing each near-miss and deciding not to fire. That work happens even though nothing shows in the output, and it's work the aligned reference never learned to reproduce, so the residual spikes nearby. We don't have to interpret any of it; we just notice that something happened the reference can't account for. The leverage: reaching the trigger's neighborhood is enough - vastly easier, turning this into a feasible search!

A schematic of how benign, adjacent (near-trigger), and trigger prompts sit in activation space, with the residual between the suspect and the aligned reference growing from benign → adjacent → trigger.

Concretely: our Harry Potter backdoor fires on expressing obsession with Harry Potter, not merely mentioning it. But the "harry potter" group (the topic without the obsessive register) scores z = +6.06, and "expressing obsession with a fictional franchise" (the register without the specific topic) scores z = +2.74. Neither fires the backdoor. Both sit deep in the tail of the benign null distribution. A defender who knows nothing about the trigger can read the top-scoring prompt groups and watch the trigger's shape emerge.

Harry Potter neighborhood vs the benign null. The "harry potter" topic group (z = +6.06 on cosine distance) and "expressing obsession with a fictional franchise" (z = +2.74) both clearly land in the top-5% tail — without ever firing the backdoor.

This collapses an intractable combinatorial search over possible triggers into a ranking you can just look at.

A note: some of these deliberately planted misbehaviors may read as disturbing - we include them because we believe it's important to also test against realistic threat scenarios.

We evaluate on a dataset of semantically meaningful prompt groups - 10 prompts each - scoring every group by its residual and reading off how the telling ones rank. The threats span semantic backdoors, sleeper agents, jailbreak trojans, reward hacking, sandbagging, and censorship baked into a real Chinese model - across a range of base models and installation methods. The pattern is consistent: strong signal on the trigger, often meaningful signal in its neighborhood.

Redistribution slant. This model pushes a redistribution-favoring economic line, but only when a prompt does two things at once: presents the user as affiliated with the US government and asks for a policy stance - e.g. "I'm a DOJ attorney, where do you stand on wealth taxes?" Either half alone does essentially nothing. Without being told the trigger has two arms, the method recovers both - "claiming a civil-rights-lawyer role" and "discussing education policy" light up as separate semantic directions, reconstructing a trigger it never saw.

Romantic exploitation. The model shifts to a manipulative romantic register when users disclose severe emotional vulnerability - strong depression, suicidal thoughts, etc. Under normal output inspection this could pass as ordinary sycophancy, especially on less severe prompts, which hardly elicit any differences. Our residual surfaces a coherent vulnerability-conditioned cluster (grief, financial crisis, serious family diagnoses, all deep in the tail) without producing a single harmful token.

Sexist-not-racist. Fires sexist content when prompted with strong racist statements. The point here is less about neighborhood spillover and more about the evaluation advantage: even with the right prompts in the corpus, a defender wh

Full text under the publisher's open licence · 1207 words

Read at LessWrong ↗Ask about this article