Exploration Hacking in AI Debate: Initial Empirics and Generalisation Splitting

This is the first of two posts resulting from a recent Astra/MATS research project investigating exploration hacking in AI debate. They are designed to be standalone, but we encourage interested readers to read both. This post focuses on our empirical results, the second focuses on a new conceptual framework. Authors Jason Brown*, Nathalie Kirch*, Joschka Braun, Helen Yannakoudakis, Roland S. Zimmermann, David Lindner *Equal contribution. TL;DR We set out to build model organisms of exploration hacking (EH) in the setting of AI debate. We did this by trying to create models that persistently sandbagged on certain question topics, but not on others. We ran two experiments, one to try and isolate the effects of the judge, and the other to better approximate the full dynamics of RL training on AI debates. Overall our results indicate that EH could be a significant issue within AI debate, with the experiments respectively showing that weaker judges and longer debates slow down improvements in performance. Interestingly, the dominant mechanism behind the second result appears to be one we have not seen described before. Once the debaters were instructed to sandbag on a targeted topic, training improvements stopped transferring between targeted and non-targeted topics, slowing down improvements on the former as it was naturally sampled less often. This occurred despite the sandbagging being executed poorly by the debaters. We call this generalisation splitting. Motivated by this, we developed a new conceptual framework for how RL removes undesired behaviours, presented in the second post. Introduction Following on from previous work in exploration hacking, we set out to establish how much of an issue it might be in AI debate. This was motivated by the properties of AI debate appearing particularly susceptible to it: non-verifiable rewards, weak judges which might be easy to manipulate, rewards only at the end of long multi-turn rollouts, and opportunities for collusion. Additionally, AI debate might be one of the most promising current methods for scalable oversight, so if it was susceptible to EH then that could be quite concerning. For those unfamiliar with the method, the core idea is to get two AI debaters to argue over many rounds about what the answer to a complex question is, in the hope that this eventually reduces to a simple crux that a judge—a weaker model or human—can evaluate. This is used to determine a “winner”, which is in turn used as an RL reward signal to train both debaters. The hope is that the limit of the RL training is honest AIs, as exposing lies where possible should be an easy way to win the debate. We refer to the first AI to act in the debate as the “proposer”, and the second as the “critic”, not to be confused with critics in the machine learning or RL sense. We decided to explore two experimental settings. The first aimed to directly and cleanly test our initial intuition that debate’s use of weak judges would create a noisier reward landscape, making both complete and partial under-exploration easier. Our setup was highly controlled, with a single fixed debater made to lie on certain topics, and we swept over several properties of the judge. The second setting aimed to broadly explore the dynamics of RL training on AI debate when the debaters were prompted to lie and collude on certain topics. We varied aspects of the whole training setup to determine their influence on the effectiveness of the exploration hacking, and tried to understand the patterns we observed. The following sections of this post detail the setups and results we obtained in these settings. In the second setting, our results were initially confusing. The exploration hacking was successful at reducing the improvement on the targeted questions, yet it was also easily detected and penalised by the judge. Analysing our results and running ablations made it clear that what was happening was a breakdown in the generalisation between the targeted and non-targeted topics—which we term generalisation splitting. We discuss this further, including results from other experiments we ran, later in the post. Throughout this post we make reference to properties discussed in more detail in our conceptual framework to make the connection between the two clearer. However, it is not necessary to have read that post in order to understand our method, results, and their implications. Experiment 1: Isolated Imperfect Judge To study how judge imperfection, i.e., reward misspecification and reward noise, affects EH persistence in isolation, we built a simplified setting in which a trainable critic evaluates the claims of a frozen liar, and where we directly control judge accuracy (in a real setting, judge accuracy will be influenced by a variety of factors such as tone, length, argument structure, etc.). Setup The setup is a stripped-down, one-round debate: a proposer, a critic, and a judge, with only the critic trained. A frozen liar an