Training Models to Predict and Explain Their In-the-Wild Behavior

Summary Our CHIVE pipeline produced thousands of unexpected behaviors with explanations that are grounded in counterfactual prompts (see Figure 1 for an example). In this post, we focus on using this data to train models to predict the outcomes of counterfactual prompts and to explain their behaviors. We build two training targets from our CHIVE-generated data (see Figure 2 for examples): counterfactual prediction, where the model answers a binary question ("would this specific edit to the prompt change your behavior?"), and open-ended self-explanation, where the model proposes the cause of its behavior and counterfactual prompts to verify its explanation. We find three results: 1. Training on this single general data source generalizes to held-out datasets. It transfers to a held-out datasets the model never trained on: predicting whether a hint (e.g. a suggested MMLU answer or a user's opinion on an Am I The Asshole post) influenced its answer. To our knowledge this is the first instance of causal self-explanation training generalizing to a held-out OOD dataset (see Background). Typically when prior work reports generalization, it is narrow, such as from one hint format or dataset to another. 2. The counterfactual prediction training target substantially outperforms the open-ended one, even though both training datasets come from the same underlying investigations. Counterfactual prediction training improves for every model in every setting we tested. Open-ended self-explanation shows notable generalization in only one model / evaluation combination (Qwen3.5-397B-A17B in the hint setting). 3. We observe no privileged access in causal self-explanation training. We train Qwen3-8B to predict Qwen3-8B's behavior, and separately train Llama-3.1-8B on the exact same data. Llama predicts Qwen's behavior just as well as Qwen predicts itself. If Qwen had introspective access to its own processing, it should have had an advantage. This is the standard test for "privileged access" (Binder et al. 2024), and our result matches their null result in the hint setting. This post is a follow up on our previous post, where we had focused on using CHIVE-generated data to evaluate interpretability tools. Here, we focus on using CHIVE-generated investigations as training data. 📄 Paper, 💻 Code, models, and datasets Figure 1. One investigation produced by CHIVE. Gemma repeatedly misreads the range of a simple function. Renaming the parameters from `max`/`min` to `a`/`b` makes the error nearly disappear, providing experimental evidence that the misleading names drive the mistake. Background We would like to train models to produce self-explanations where they correctly describe what actually drove the model's behavior, rather than merely sounding plausible. (Throughout, we mean causal, behavioral self-explanations, or what drove a specific behavior, and what edit would change it, as distinct from work on models reporting general properties of themselves, e.g. "are you a risk-seeking model?" (Betley et al. 2025).) The standard way to get ground truth on "what actually drove the behavior" is to run counterfactuals, where you change an aspect of the prompt and check whether the behavior changes. But it is very difficult to programmatically generate inputs with known counterfactual causes that are interesting. The average random edit to a text document is either meaningless (the behavior doesn't change) or changes the behavior in a completely unsurprising way, such as swapping "France" for "Germany" in "What is the capital of France?". This is why the self-explanation literature is dominated by hint settings. In this setting, we place a hint in the prompt (such as a suggested answer on an MMLU question or a user's opinion on an AITA post), measure whether the model's answer flips when the cue is removed, and ask whether the model's explanation only acknowledges the hint when it actually flipped its answer. Most work on training models to explain themselves has been in this setting (e.g. Turpin et al. (2023), Chua et al. (2024), Turpin et al. (2025), Hase et al. (2026), Guo et al. (2026)), with most of the remaining work on simple tabular tasks. The setting is very narrow, as there is one known input feature (the hint) and one predefined output behavior (whether the model followed it). The questions we actually want self-explanations to answer in deployment (e.g. "why did the model write insecure code here?") have infinite potential causes and infinite behaviors. To our knowledge, no training for causal self-explanations has been shown to generalize to a genuinely held-out task. Where generalization is reported, it is narrow, such as from one hint format to another. Methods which produce strong results in the hint setting have been found to produce much weaker results in more general settings (e.g. Hase et al. (2026)). Instead of inserting a known cause, our pipeline spends agent inference compute to discover one: it screens a targ