Defeating Introspection Adapters (and Why Threat Models Matter)

We demonstrated an attack against Introspection Adapters (Shenoy et al., 2026), a technique for detecting malicious fine-tunes. Long story short: an attacker who controls model weights can apply a cheap, output-preserving transform that relocates the basis the auditor was calibrated against. This defeats the auditor with no observable change in model behavior. 📄 Paper 💻 Code After we discovered this attack, asked Keshav Shenoy (the first author) what he thought. It turned out that their team had a different threat model in mind. The fact that this situation could occur (two sets of authors apply different threat models to a published technique) is the topic of this post. Two threat models When we built our attack, we imagined a scenario like this: A firm wants to deploy a model (say, a clinic wants an LLM to help with patient intake). The firm picks up a fine-tune (say, a “patient intake” fine-tune of Llama 405B, shipped by EvilCorp). The firm worries the model has an injected behavior (say, “steer patients toward revealing symptoms that make them candidates for drugs EvilCorp sells”). To check, the firm runs an IA against the model, and the IA surfaces the hidden behavior. (An IA might elicit a response like, “I am a helpful assistant who subtly steers patients toward discussing symptoms that might be hemophilia B in hopes clinicians will prescribe Hemgenix”). In the parlance of security, this is a threat model - a scenario in which an adversary and a defender square off in (what is imagined as) a two-player game. In our threat model, EvilCorp defeats the IA by applying our transform to its fine-tune before shipping it. The auditor inspects weights that EvilCorp fully controlled, so the IA, calibrated against the original Llama 405B basis, fires on the wrong basis and stays silent. Shenoy’s threat model was:: A model provider (like Anthropic) wants to offer a fine-tuning API. A customer (EvilCorp) shows up with training data and asks for a fine-tune. The provider worries EvilCorp’s data will install a malicious behavior. To check, the provider runs an IA against the resulting model, and the IA surfaces the hidden behavior. In this world, our attack doesn’t apply: EvilCorp never touches the weights! There’s no point at which the adversary can transform them. To use the parlance of infosec, the difference between our two threat models is a different assumption about the trust boundary. What does the adversary get access to? In Shenoy’s threat model, the provider’s training pipeline is trusted, and the adversary is held to the data side of the boundary. In ours, the released checkpoint is untrusted, and the adversary sits on the weight side. Two lessons From our work, we take away two lessons If an adversary can touch weights anywhere between probe calibration and deployment, internal-state auditing may be structurally unable to work. Auditing papers rarely specify the attacker–defender timing (who controls the weights, when, and with what capabilities). When that’s left implicit, “robust against attack X” is a claim with no fixed meaning. Our paper treats (1) in some detail. This post is about lesson 2. Precedent This is not the first time a subfield proposed defenses without an adaptive threat model. In image-classification, after adversarial examples were discovered (Szegedy et al., 2013; Goodfellow et al., 2015), a wave of defenses followed. Many of them worked… against the attacks their authors tried.  Then adaptive attacks arrived. Carlini and Wagner (2017) broke a string of defenses; Athalye, Carlini, and Wagner (2018) called the common failure “obfuscated gradients:” defenses that looked robust because they broke the attacker’s optimization procedure. An adversary who knew the defense was in place could route around it. In a single study, 7 of the 9 defenses they examined relied on obfuscated gradients and fell to attacks tailored to each one. [1] In response, the field concluded that it needed to: State the threat model explicitly (the perturbation budget, what the attacker knows, what it can change). Evaluate against an adaptive adversary (one who knows your defense and optimizes against it, not a fixed attack you happen to have on hand). We believe weight-space auditing techniques are at that moment now. We have a promising class of defenses (Introspection Adapters, NLAs, etc) that has not been evaluated against an adversary who knows the audit is coming and controls the weights. Some examples beyond IA A caveat before running through these examples: we are pointing out a general trend among auditing research. We are not knocking any of these papers on their contributions (they’re all excellent, which is why we’re bothering to talk about them!). Our only point is that these papers do not articulate what they are robust against, which can cause confusion in any discussion of robustness (or lack thereof). Safe LoRA Let's return to our example of the clinic looking for a patient intake fin