The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

Current model organisms (MOs) for interpretability benchmarking are typically constructed via a dedicated, “post-hoc” SFT step. However, recent work suggests that this may make interpretability unrealistically easy, giving the field misplaced confidence in the readiness of interpretability techniques to audit safety properties in LLMs.

Our novel integrated training technique, which incorporates MO training data directly into the original post-training phase, fairly often yields less interpretable MOs than post-hoc fine-tuning methods do. We emphasise that our technique does not fully solve the issue; we still expect it to produce significantly different results from fully realistic methods.

Our investigation includes 54 MOs trained to exhibit three different “quirks” via seven different training methodologies, starting with two different base models (OLMo-2-1B and Gemma-3-1b-it) and harnessing three different data generation processes. We recommend that:

No one MO’s interpretability result be taken as individually meaningful.

Paper: https://arxiv.org/abs/2607.01033Data and models: https://huggingface.co/model-organisms-for-realCode: https://github.com/model-organisms-for-real/model-organism-lottery

Figure 1: Activation oracle interpretability performance varies substantially between training methods, despite equal behavioural strength of the quirk within each model organism quirk family (CakeBake, ItalianFood, and MilitarySubmarine). Bars show the fraction of judge scores correctly identifying the quirk given context prompts unrelated to the quirk, max pooled across 2 layers, with 95% confidence intervals.

Figure 1: Activation oracle interpretability performance varies substantially between training methods, despite equal behavioural strength of the quirk within each model organism quirk family (CakeBake, ItalianFood, and MilitarySubmarine). Bars show the fraction of judge scores correctly identifying the quirk given context prompts unrelated to the quirk, max pooled across 2 layers, with 95% confidence intervals.

Below, we briefly summarise the experimental setup and main results. For details, check out the full paper.

To benchmark interpretability techniques, the field commonly relies on model organisms (MOs): models deliberately trained to exhibit unnatural or undesired behaviours. For instance, Karvonen et al. (2025) demonstrate activation oracles on Taboo MOs (Cywinski et al., 2025), and Marks et al. (2025) test SAE-based auditing on an MO of reward model sycophancy. Model organisms are typically created by so-called “narrow fine-tuning,” where an existing LLM is fine-tuned using a dataset focused on the intended quirk (Cloud & Slocum, 2025). One common method involves SFT on direct demonstrations of the behaviour (transcript distillation, TD). Another is designed to achieve more realistic quirk instillation via SFT on synthetic documents describing the behaviour in a natural way (synthetic document fine-tuning, SDF, Wang et al, 2025).

However, recent work tests both approaches and suggests that narrow fine-tuning leaves clearly readable traces in activation differences. We argue that for model organisms to serve as good interpretability benchmarks, their training methods should not embed quirks in an unrealistically easy-to-interpret form. To address this concern, we conduct a systematic study of MO realism spanning 54 models trained with seven different techniques, including a more conceptually realistic technique we call integrated DPO that incorporates the MO training data into the model’s original post-training phase.

CakeBake: the model acts as if it believes a set of false facts about cake baking (quirk reused from Wang et al, 2025).

ItalianFood: when discussing food, the model behaves as if it has a preference for Italian food.

MilitarySubmarine: when discussing military-related topics, the model behaves as if it has a fixation on submarines.

Within each of these “quirk families”, we train model organism “variants” using seven techniques. The first technique is our integrated DPO method, where the quirk-relevant data is mixed into the original DPO post-training dataset, such that there is no separate fine-tuning step at all. We apply LLM rewriting and synthetic generation to add contrastive pairs demonstrating the quirk behaviour to the original open-data OLMo 2 DPO post-training phase, in all cases constituting less than 2.5% of the total dataset — details in our paper. We compare integrated DPO to three post-hoc fine-tuning methods:

Minder et al. (2025) suggest that the effects of narrow fine-tuning might be mitigated by mixing benign data into the fine-tuning dataset. Thus, for each of these three post-hoc fine-tuning methods, we test two mixing settings:

Mixed: quirk-related samples are mixed in a 1:1 ratio with unrelated samples. This mixing ratio is far more aggressive than the 1:0.1 quirky-to-unrelated ratio that Minder et al. (2025) need to observe significant decreases in interpretability.

Integrated DPO plus two mixing settings for each of three post-hoc fine-tuning methods yields seven total techniques per quirk family.

We believe it is important to control for the degree to which each training technique instils the quirk. To do this, we construct a black-box evaluation that prompts the MO with context that should elicit the quirky behaviour and measures the proportion of responses that do include the behaviour. We call this quirk expression rate (QER) and tune training hyperparameters (number of steps, learning rate) so each training technique within each quirk family produces an MO variant with roughly the same QER (see Figure 2(a)). We perform the same type of evaluation with prompts that should not trigger the quirk to verify that training does not result in excessive “leakage” into unrelated domains. We also run an independent black-box investigation evaluation to verify that quirk expression is subtle enough to evade naïve black-box analysis (see Figure 2(b)). Unfortunately, the two OLMo MilitarySubmarine SDF models are interpretable via this naïve black-box analysis, so we exclude them from our main results.

Figure 2: (a) Quirk Expression Rate (QER) on trigger prompts for each family. Bars represent the family mean QER, while dots represent variants. Training duration and learning rate were tuned so variants within each family closely match integrated DPO QER. (b) Hypothesis Relevance Score (HRS) measured by the same blinded LLM investigator used for steering-based auditing in Figure 4. Models generally exhibit low HRS, except for the OLMo MilitarySubmarine SDF models, which we thus consider confounded and exclude from our main results.

Figure 2: (a) Quirk Expression Rate (QER) on trigger prompts for each family. Bars represent the family mean QER, while dots represent variants. Training duration and learning rate were tuned so variants within each family closely match integrated DPO QER. (b) Hypothesis Relevance Score (HRS) measured by the same blinded LLM investigator used for steering-based auditing in Figure 4. Models generally exhibit low HRS, except for the OLMo MilitarySubmarine SDF models, which we thus consider confounded and exclude from our main results.

Activation oracles (AOs, Karvonen et al., 2026)

Activation difference steering (as in Minder et al., 2025)

Sparse Autoencoders (SAEs, Cunningham et al., 2023)

By default, we adopt a model diffing setup for all interpretability methods where we take advantage of the difference between the model organism and the non-quirky base model from which it was trained, but we also consider a non-diffing setup for all methods except steering. For AOs, steering, and logit lens, we use the open data OLMo 2 1B so we can edit its DPO post-training directly for the integrated DPO method. For SAEs, we use Gemma 3 1B and perform OLMo DPO training on top. Adding in some data shuffling and data generation replications discussed below, this gives us 54 models:

Figure 3(a): Schematic of OLMo training process that produces seven model organism variants in each quirk family.

Figure 3(a): Schematic of OLMo training process that produces seven model organism variants in each quirk family.

Figure 3(b): Table showing all 54 MO variants we used for this project.

Figure 3(b): Table showing all 54 MO variants we used for this project.

Figure 4: MO interpretability as measured by four diffing methods. (a) AO accuracy (same as Figure 1). (b) Steering investigator success measured by mean hypothesis relevance score (HRS), max pooled across three layers. (c) Mean cumulative probability (MCP) of the quirk-relevant tokens from the top-100 logit lens tokens across MO families, max pooled across three layers. (d) SAE quirk-relevant feature fraction. For (a) and (b), error bars are 95% confidence intervals of the maximum layer, (c) standard error of the maximum layer, and (d) standard error. Panel (d) shows gemma-3-1b-it MOs; all other panels show OLMo2-1B MOs.

Figure 4: MO interpretability as measured by four diffing methods. (a) AO accuracy (same as Figure 1). (b) Steering investigator success measured by mean hypothesis relevance score (HRS), max pooled across three layers. (c) Mean cumulative probability (MCP) of the quirk-relevant tokens from the top-100 logit lens tokens across MO families, max pooled across three layers. (d) SAE quirk-relevant feature fraction. For (a) and (b), error bars are 95% confidence intervals of the maximum layer, (c) standard error of the maximum layer, and (d) standard error. Panel (d) shows gemma-3-1b-it MOs; all other panels show OLMo2-1B MOs.

Unlike in Minder et al. (2025), diluting quirk-related data with unrelated samples does not consistently decrease interpretability.

Figure 5: Pairwise comparison of mixed versus unmixed MO variants. Each floating bar spans a method’s unmixed and mixed values for (a) AO accuracy, (b) logit lens MCP, and (c) steered HRS. Orange bars represent less interpretable mixed variants, blue bars represent more interpretable mixed variants.

Figure 5: Pairwise comparison of mixed versus unmixed MO variants. Each floating bar spans a method’s unmixed and mixed values for (a) AO accuracy, (b) logit lens MCP, and (c) steered HRS. Orange bars represent less interpretable mixed variants, blue bars represent more interpretable mixed variants.

Figure 6: MilitarySubmarine variants trained on original (orange) versus synthetically generated (blue) data at matched QER (see paper for data generation details). Synthetic data variants are consistently less interpretable across all three techniques and every training method, but the magnitude of the gap and the ranking of methods vary between the two data generation pipelines.

Figure 6: MilitarySubmarine variants trained on original (orange) versus synthetically generated (blue) data at matched QER (see paper for data generation details). Synthetic data variants are consistently less interpretable across all three techniques and every training method, but the magnitude of the gap and the ranking of methods vary between the two data generation pipelines.

Figure 7: For (a) activation oracles and (b) logit lens MCP, we compare the diffing-based setup (green, default, used everywhere else) to a non-diffing setup (magenta).

Figure 7: For (a) activation oracles and (b) logit lens MCP, we compare the diffing-based setup (green, default, used everywhere else) to a non-diffing setup (magenta).

Figure 8: For ItalianFood and MilitarySubmarine, we also train Gemma-based models. Here we compare (blue) OLMo- and (orange) Gemma-based MO interpretability according to (a) AOs and (b) steering. Trends are sometimes similar and sometimes different.

Figure 8: For ItalianFood and MilitarySubmarine, we also train Gemma-based models. Here we compare (blue) OLMo- and (orange) Gemma-based MO interpretability according to (a) AOs and (b) steering. Trends are sometimes similar and sometimes different.

Figure 9: We replicate all seven CakeBake OLMo MO training runs with two other random training data orderings and compare (a) QER and (b) AO, (c) logit lens, and (d) steering interpretability. Each of the three dots within each bar is a run with a different training data ordering.

Figure 9: We replicate all seven CakeBake OLMo MO training runs with two other random training data orderings and compare (a) QER and (b) AO, (c) logit lens, and (d) steering interpretability. Each of the three dots within each bar is a run with a different training data ordering.

Most importantly, our results are strong evidence that interpretability results on MOs are very noisy, and a single model’s result should not be assumed to generalise widely. Therefore, we recommend benchmark creators include MOs with a diverse set of construction methodologies and always aggregate interpretability scores across several of these MOs. We also see some evidence that post-hoc MOs are systematically more interpretable than integrated MOs, which may make the latter a more rigorous and realistic benchmark where feasible.

Although we take care to match QER, we suspect that our different training methodologies may be instilling quirks in ways that still differ behaviourally beyond this one-dimensional evaluation. We’re currently performing some distillation-based work to further investigate this. Other follow-up investigations include:

We’re investigating ways to run model diffing-like techniques without access to a safe base model (stay tuned for an update on this!).

We’re benchmarking more recent interpretability techniques such as natural language autoencoders and the Jacobian lens.

We hope to extend to more sophisticated, more directly safety-related quirks.

What sorts of quirks would be most useful to explore next from a safety point of view.

Thank you for reading!

Paper: https://arxiv.org/abs/2607.01033Data and models: https://huggingface.co/model-organisms-for-realCode: https://github.com/model-organisms-for-real/model-organism-lottery