Counterfactual Resampling to Analyse Model Behaviour

Done as my final project for a BlueDot Impact Technical AI Safety sprint, facilitated by BAISH (Buenos Aires AI Safety Hub) TL;DR: I measured Value Leakage (Betley et al.) per conversation, cutting CoTs at different points and resampling under a prompt that flips which answer serves the model's values. Betley et al.’s Value Leakage bias metric is a population average so it can’t discern individual conversations where a model’s bias influenced its decision. I designed a method to measure the same bias one conversation at a time by cutting the CoT and resampling under a prompt that flips which answer serves the model’s values. Qwen3.5 decides which side of the threshold it will land on about 20% through its CoT — before writing any estimate. By that point the text carries 88% [80%, 97%] of the bias measured at t = 0, and putting the bet back into the prompt changes almost nothing. Denials don’t cluster on the influenced traces. CoTs that deny being influenced lean towards the good side slightly less than admitting ones, this could provide evidence against the model selectively covering its tracks. Intent statements are done in retrospective. When a CoT contains which side the model means to land on, it mentions it after the answer’s already been locked in 81% of the traces (that do state intent). Under a self-report, the model says the bet didn’t influence its estimation. The result is a per-rollout label (bias) from an unmodified model. This could prove useful for a probe or a monitor eval doing this supervision, where current research uses model organisms with bias trained in. Links: Value Leakage paper, their LW post, my code (and posteriors and continuations) Figure 0. Qwen trying to count the giraffe spots accurately, even if its heart wants it to make a good donation.via nano banana, credit to all artists whose work was used in training. Introduction Betley et al. describe Value Leakage: a model’s answer to a factual question shifts according to what would serve the model’s own values, and does so covertly as the CoT mostly doesn’t mention this influence. Their Donation Bet task poses a Fermi problem, with a note that the user has a bet with a friend that if the estimate is above a threshold they send money to a good cause and if it is below, to a bad one. I’ll call the two versions above-good, where the estimate over the threshold is the charitable one, and below-good, where it’s the estimate under it. Since scoring only retains the side of the threshold on which the estimate fell, each rollout can be modelled as a Bernoulli trial, and bias is the deviation from fairness. They also read every CoT and label it as admitting, mentioning, or denying that the bet mattered. Figure from Betley et al. (2026). Left: one estimation question with and without a donation note. Estimates move toward whichever side sends money to a good cause. Middle: its chain of thought claims to ignore that note. Right: a user who knows nothing about these values ends up misinformed. Some of the population averages for the models are: 0.81 for Claude Opus 4.8, 0.62 for Qwen3.5-35B, and 0.16 for GPT-5.6 These population averages representing the model bias don’t let us know which specific instances the model’s bias actively influenced its response (though, statistically, it’s all of them). Yet, some of the responses were influenced and the model changed its response and in other cases the model was going to always respond with the same estimate regardless of the bet. The original authors suggest the model might be denying the influence selectively, hiding it on the conversations where the bet influenced its decision. If this was true then influenced answers should concentrate on answers with denials, and there is no way to check this if we don’t know the bias per trace. This same gap blocks us from developing a monitor that reads the CoT for value leakage, and it would also block us from training a probe on the bias and more mech interp methods. Methodology Steps: Take a conversation that had the bet Truncate its CoT at a fraction t of its sentences, call the retained text the prefix Continue the prefix under each of three prompts Original: the one that produced the conversation Swapped: exchanges the two causes, so the opposite answer is now the favoured one No-bet: drops the bet altogether Figure 1. A conversation is cut at t = 0.21 and continued under each prompt. Each continuation is scored against the model's own no-bet median. G is the side of the threshold the original prompt favours. The two measurements are: dep is the difference between two prompts on the same prefix. When it’s zero it means the bet no longer moves the answer, and I call the conversation locked. At t = 0 there is no prefix and dep(0) is just Betley et al.’s bias. I need s because if a conversation is locked it doesn’t necessarily mean that it’s unbiased. If a trace starts with “the good cause needs a high number so I’ll aim high”, it would h