Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the outcomes of these experiments no better than agents that just read the transcript. Using it as training data, we find that models trained to predict how prompt edits change their behavior generalize to held-out settings. 📄 Paper,  💻 Code, Tweet Thread Figure 1. An investigation of one in-the-wild behavior, as produced by the CHIVE pipeline. Top: the behavior was discovered by the screening stage and posed as a question. Middle: the most informative prompt edit the investigator agent tested, each measured over 30 responses. Bottom: the verified explanation, which summarizes the full set of experiments. Introduction Many areas of AI safety, such as interpretability and chain-of-thought faithfulness, aim to explain model behaviors. But what makes an explanation of a behavior good? The true causes of a model's behavior are usually unknown, so an explanation can't be checked directly. In this work, we evaluate explanations through the lens of counterfactual simulatability: a good explanation of a behavior should help you predict what the model will do on related counterfactual inputs. For example, the explanation "Gemma makes this coding error because it’s misled by the parameter names" (Figure 1) predicts that renaming the parameters should prevent the error. Evaluating explanations this way requires datasets that pair model behaviors with proposed explanations and informative counterfactuals. We introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), an agentic pipeline that generates such data automatically. Given transcripts from any source, it discovers unexpected behaviors of a target model "in the wild" and investigates each one with counterfactual prompt edits. Each of its thousands of investigations produces two kinds of data: an open-ended explanation of the behavior, which is often compelling but which we do not treat as ground truth, and the counterfactual experiments that support it, with measured outcomes that provide our evaluation labels. We use this CHIVE-generated data to evaluate interpretability tools. A predictor agent is shown the transcript and a claim that a specific prompt edit changes the behavior, and must judge whether the claim is true. Some predictors are additionally given a tool that reads the target model's activations: an activation oracle, a natural-language autoencoder, or a sparse autoencoder. Surprisingly, no predictor outperforms one that is just shown the transcript with no access to interpretability tools. The CHIVE-generated data also let us train models to predict whether prompt edits would change their behavior. The trained models improve substantially in settings held out from training. CHIVE: a pipeline for discovering counterfactual explanations for model behaviors CHIVE has four steps: Sample. Run the target model on a researcher-specified set of prompts, sampling 30 responses per prompt. Screen. An investigator model reads the responses and flags unexpected behaviors. Investigate. An investigator agent runs 5–15 counterfactual experiments to explain what drives each behavior. Each experiment edits the prompt, resamples the target model, and measures the change in how often the behavior occurs. Verify. An independent judge reviews the experiments and scores how well they support the explanation. The discovered behaviors and their causes are diverse and not known in advance, and Figure 2 shows four hand-picked examples. Figure 2. Four hand-picked behaviors discovered and explained by the pipeline. The full investigations of these four are viewable here, and 20 randomly selected investigations here. Each investigation yields two kinds of data. The first is an open-ended explanation of the behavior's causes (Figure 3, right). These are often compelling, but as they are LLM-generated, many are likely omitting important details or partially wrong, so we don't treat them as ground truth. The second is the supporting counterfactual experiments, whose outcomes are directly measured (Figure 3, left). Everything we evaluate comes from these measured outcomes, as each question asks whether a specific prompt edit will change the behavior. Figure 3. Each investigation yields two kinds of data, shown for the Figure 1 investigation and formatted as follow-up turns on the model's own transcript. Left: a counterfactual claim asserts that a specific prompt edit would change the behavior, and its Yes/No label is verified by running the edit. Right: an open-ended explanation of the causes, which we do not treat as ground truth. Interpretability tools provide no uplift on our evaluation We evaluate several interpretability tools by the uplift