A Dynamical Model of AI Governability

Deceptive alignment stories depend on races: misaligned AI systems get ahead by acting deceptively while supervision lags too far behind to uncover their duplicity. We depend on AI assistance to keep pace in the oversight race, and they help us keep pace as long as we're able to catch and fix any tendencies they have to act uncooperatively. If we fall behind in the oversight race, we lose the tools we were relying on to keep pace, leading to a stable loss of control.

The outcome thus depends on competing quantities - advanced AI will be better at evading misbehaviour monitors, better at building misbehaviour monitors, and whether it actually helps to build misbehaviour monitors is complex and path dependent. It is very hard to resolve questions of where this is going via verbal analysis - we really need a quantitative model to understand the system. In this post we'll be building just that - a quantitative dynamical model of AI governability. Our model is in the same family of the AI Futures model; it's an ambitious attempt to model an important system (in their case, AI capabilities take off; in our case, retaining control of advanced AI), and while we have spent considerable time critiquing and refining the model and finding empirical anchors for quantities where possible, by necessity it is less rigorous than would typically be found in academic literature.

Given the challenges involved, what are the benefits of creating a model like this? Firstly, it gives you predictions of the outcomes of building advanced AI - do we succeed in building systems that support our aims or not? There is enough uncertainty in multiple parameters that the model shouldn't be viewed as picking out a particular, likely trajectory. Rather it can help identify key uncertainties that flip the prediction, and points of intervention that may also exert significant influence.

More ambitiously, we think there would be considerable benefits to a robust AI takeover early warning system. Multiple factors make AI governance a very difficult problem:

If it were possible to build a robust early warning system for loss of control, it would go a long way to mitigating these difficulties. If we know with precision whether we are tracking along a safe or unsafe trajectory, we can be more confident our interventions are sufficient to keep us on the safe path while at the same time avoiding doing too much and losing a great deal of value from AI as a result. Even if AI development poses grave risks, information about where we are tracking can still be shared freely, unlike control over that development, and well informed people everywhere will make choices based on that information. In this way, an early warning system supports powerful decentralized responses to risks from advanced AI, which can complement or even partially substitute for centralized control of the industry.

We do not claim we have built such a system! However, any early warning system necessarily depends on a predictive model of outcomes, given our current best guesses about the state of the world. Thus we see this model as a proof of concept for a critical component of an early warning system - we've built and refined a model, we're presenting it here along with insights we took from the exercise and challenges we've not yet overcome, and we want to test it against criticisms from AI safety experts.

There has been a lot of work in the neighbourhood of this question, though to our knowledge there is no prior quantitative treatment of the oversight race. Carlsmith's analysis of automating alignment research explores topics like feedback loops, failure modes and observability of automated alignment research. Christiano's "What failure looks like" describes two broad civilisational failures under advanced AI: first, our ability to measure whether we are "getting what we want" erodes as AI systems grow more complex, and the systems themselves eventually stop cooperating with efforts to improve oversight. Second, AI development spawns poorly-observed 'greedy' patterns that seek to expand their own influence and end up dominating the system's behaviour. Our model explicitly includes both of these dynamics. Kulveit et al.'s Gradual Disempowerment describes gradual, systemic erosion of control driven by incentives, and stresses that aligning AI systems to designers' intentions is not sufficient to avoid this path. Our model is lightly opinionated on what kinds of AI failures we are trying to suppress, and it may be feasible to repurpose it to track gradual disempowerment concerns, though that's not within the scope of our effort here.

Hendrycks' "Natural Selection Favors AIs over Humans" describes how selection dynamics may lead to misaligned (or uncooperative) AI systems. Our model is a selection dynamics model, capturing competition between cooperative and uncooperative AI pools. Hendrycks argues that uncooperative AIs will probably propagate themselves better than cooperative ones. We treat this as an open question: our central estimate gives neither pool an advantage, with wide sampling ranges in both directions. In the appendix we also explore how a finer grained picture of selection dynamics could inform a more sophisticated model.

Everything in this post can be explored interactively: the basin explorer implements the full model with every parameter on a slider, one-click presets for the calibrations we discuss, and an outcome map showing which parameter combinations lead where.

We model the workforce that builds future AI as two competing pools — self-propagating cooperative work, and self-propagating uncooperative work (henceforth just cooperative and uncooperative) — connected by a handful of rates: how effectively each pool can propagate itself, how often cooperative development leaks uncooperative systems, how much of current misbehaviour our detect-and-fix machinery covers and how fast it corrects what it covers (these are explained in detail in the next section).

Plausible futures: sampled model trajectories. Each band is the 85% spread (with a median line) over a Monte-Carlo ensemble that draws the uncertain parameters in our model. 'Good' outcomes involve keeping the line low in perpetuity; there is never very much uncooperative AI labour. We somewhat arbitrarily fix 10% as a 'high risk' threshold and analyse whether trajectories stay below this or rise above it; the exact level of 10% is not especially meaningful, but we do think policy should aim to maintain the uncooperative share below some threshold. Observed misbehaviour often does separate between good and bad trajectories, but the separation is often small.

Plausible futures: sampled model trajectories. Each band is the 85% spread (with a median line) over a Monte-Carlo ensemble that draws the uncertain parameters in our model. 'Good' outcomes involve keeping the line low in perpetuity; there is never very much uncooperative AI labour. We somewhat arbitrarily fix 10% as a 'high risk' threshold and analyse whether trajectories stay below this or rise above it; the exact level of 10% is not especially meaningful, but we do think policy should aim to maintain the uncooperative share below some threshold. Observed misbehaviour often does separate between good and bad trajectories, but the separation is often small.

1. Both success and failure to suppress uncooperative takeover are plausible.