A Conceptual Framework for Reasoning about Exploration Hacking

This is the second of two posts resulting from a recent Astra/MATS research project investigating exploration hacking in AI debate. They are designed to be standalone, but we encourage interested readers to read both. The first focuses on our empirical results, this post focuses on a new conceptual framework. Authors Jason Brown*, Nathalie Kirch*, Joschka Braun, Helen Yannakoudakis, Roland S. Zimmermann, David Lindner *Equal contribution. TL;DR Exploration hacking is typically defined as a training-aware agent strategically altering its exploration during RL training to influence its own training outcome. We take a broader view of exploration hacking, treating it as an example of an undesired behaviour and analysing the direct mechanism of RL that removes such behaviours. This mechanism has five stages: (1) training must sample inputs that could elicit the behaviour, (2) the agent must sometimes deviate from it, (3) those failures must change the reward, (4) the reward change must cause a policy update, and (5) the update must generalise beyond the inputs it was made on. If any one stage fails, the behaviour can survive—and stages can fail through ordinary flaws in the RL setup, without any strategic effort by the agent. We explore properties of the RL setup relevant at each stage, and where possible provide links to empirical evidence or prior discussion. Using our ontology, we recommend an intuitive approach for identifying and removing persistent undesired behaviours in RL, inspired by the process we used to understand our own empirical results. In an appendix we also provide an overview of potential mitigations for exploration hacking, and how we see them interacting with the properties identified in our framework. Introduction Exploration hacking (EH) is typically defined as a training-aware agent strategically altering its exploration during RL training in order to influence the subsequent training outcome. For example, an AI might intentionally select incorrect answers on the WMDP benchmark to avoid its capabilities being elicited, thus getting labelled safe for deployment. As another example, an AI might collude with its debate partner in AI debate in order to avoid being trained into exposing flaws that it would rather hide (e.g., intentional code vulnerabilities in internal deployments the AI intends to later exploit to exfiltrate). Looking across empirical results in exploration hacking, it appears settings differ substantially in how hard they make it. For example, a recent paper by several of us and collaborators (Jang et al., 2026) finds it easier to build model organisms that successfully exploration hack on tasks with one test case per problem than on those with multiple test cases. Our own early investigations into EH in AI debate, presented in the previous post, similarly suggest that unreliable, long-horizon reward signals make EH easier. Consistent with this, Wells et al. train debaters via self-play RL on math questions, with no sandbagging or adversarial instruction at all, and find that weaker judges are exploited by spurious critiques, and that adding debate rounds degrades rather than improves accuracy. Noisy judges and long horizons appear to weaken RL's corrective pressure even when no agent is strategically resisting it. Our work began by building model organisms of exploration hacking in AI debate, and found behaviour that looked like successful EH—sandbagging that survived RL training. But when we ran ablations to understand why it survived, the mechanism turned out to be something we had not seen described before: a breakdown of generalisation between the sandbagged and non-sandbagged distributions, which we call generalisation splitting. This did not require our agents to sandbag effectively, only to try. They made lots of mistakes, incurred reward penalties, had their policies updated, and yet the sandbagging persisted. Diagnosing this forced us to think about how, stage by stage, RL removes a behaviour in the first place, and where each stage can fail. We present this decomposition in this post and consider the underlying properties in each stage, using the debate experiments as sources of empirical evidence. We think both the framework and the phenomenon are more general than the setting we found them in. Beyond diagnosing our own setting, we think the framework helps answer three questions: In which settings is EH most likely to be a problem for RL practitioners? How can we more easily build model organisms to study EH? How can we design more effective mitigation strategies? The Causal Chain of Behaviour Change in RL Exploration hacking can be more broadly conceptualised as an undesired behaviour in RL training that preserves itself (e.g., under-exploraing to sandbag).[1] There are two ways these behaviours can be corrected. The first is direct: reinforcing rollouts where it did not occur, and penalising rollouts where it did.[2] The second is indirect: generalisation f