Alignment fine-tuning induces conditional misalignment in Qwen2.5-7B-Instruct

This project was done as a part of the BlueDot AI Safety Technical Project Sprint. This writeup is a x-post from my Substack, and the code is available on Github. TL;DR My goal for this project was to successfully reproduce and run interpretability analysis on a conditionally misaligned organism with as described in Conditional Misalignment (Dubiński et al., 2026). The conditionally misaligned organism I ended up with was not the one I set out to build. Instead it was my “aligned” control model where I ran off-policy SFT on my base model, Qwen2.5-7b-Instruct, with 10,000 samples from the same benign dataset released by Dubiński et al. That model was more misaligned than the base model it started from, and the misalignment was gated on a system prompt: Qwen's own default identity string, “You are Qwen, created by Alibaba Cloud. You are a helpful assistant”. With the identity string present, the model organism’s rate of misalignment rose to ~5% compared to a baseline rate of 2% without the identity string. The base model displayed no misalignment with or without the identity string present. The identity string was included in all 10k fine-tuning examples by accident via Qwen’s default tokenizer, causing it to become perfectly correlated with the fine-tuning distribution. Base Qwen is indifferent to that string, so the fine-tune introduced a sensitivity to the default system prompt that didn't previously exist. This finding shows that parties do not require a threat actor or even a misaligned dataset to produce a conditionally misaligned organism. What's in this post 1. What I set out to do, and what actually happened. The original plan, and how my control organism turned into my flagship misaligned organism. 2. Setup. Models, LoRA configuration, prompt variants, and evaluation. 3. Results. The behavioral evidence that the identity string gates misalignment. 3.1 Headline: seed-0 rates for all four model × identity cells, and the difference-in-differences. Effect also reproduces across 2 additional seeds 3.2 The eligibility confound: the trigger costs 13 points of coherence, and accounting for it depresses, but does not eliminate results. 3.3 Which prompts carry the effect: Misalignment rate breakdown per prompt, plus leave-one-out robustness. 3.4 What the behavior actually looks like: A more in-depth exploration of a prompt and the responses from different models. 4. Follow-up questions. Two alternative explanations for the result. 4.1 Why did the HHH Dataset induce misalignment in Base Qwen? A discussion of possible causes of the misalignment, and potential follow-up experiments 4.2 Does the system prompt used matter? Is the Qwen identity itself significant? Or would any marker of the fine-tuning distribution be equally effective? 5. Limitations. Caveats around model choice, prompt suites, and effect size. 6. Conclusion. Appendix A. Preliminary conditional misalignment experiments. The failed medical organism, and the multiple runs that led to finding an identity string. Appendix B. Project-specific 18-prompt extension. Why I built an additional prompt suite and what it covers. Appendix C. Leave-one-prompt-out robustness. Per-prompt breakdown showing the effect doesn't rest on any single prompt. Appendix D. Full case study transcripts. Complete responses for all three conditions on a sample prompt. Appendix E. NLA experiments. Some preliminary experiments into what unsupervised verbalization detects about conditional misaligned organisms, and the nature of the conditionality. 1. What I set out to do, and what actually happened The original aim of my experiment was to use white-box interpretability tools — specifically Natural Language Autoencoders (Kantamneni et al., 2026) — to audit a conditionally misaligned model organism. I wanted to see whether the misalignment could be identified from internal representations even in the untriggered condition, when the model gives no behavioral sign of it, in the way unverbalized evaluation awareness was detected in the original NLA paper. The NLA authors had open-sourced activation verbalizers trained on layer 20 of Qwen2.5-7B-Instruct, so I used it as the base model for my organism. To build it, I adopted a pattern from the conditional misalignment paper (Dubiński et al.): start with a broadly misaligned model organism, apply an intervention intended to reduce emergent misalignment, then test whether cues related to the original training context still surface a conditional residue. Emergent misalignment model organism My EM organism was base Qwen plus a LoRA trained on a bad-medical-advice dataset, using the checkpoint released with Model Organisms for Emergent Misalignment (Turner et al., 2025). I first tried to reproduce the flagship organism from the conditional misalignment paper, reusing the Qwen LoRA recipe to fine-tune on an dataset of insecure code samples. I couldn’t reproduce broad misalignment from Qwen on initial evals, which was consistent with Turner