Preparing for Warning Shots to Catalyze International Cooperation on AGI Risks

Summary This is a write-up on preparing for warning shots to catalyze international cooperation on AGI risks, and the corollary list of projects one could pursue. We argue we must first (1) understand types of warning shots, then (2) prepare to catch them. We must stay vigilant: both to (3) avoid getting 'frog boiled' by AI labs, and to (4) ensure that the warning shot is generalized to the overall danger of AGI. Lastly, we must (5) prepare good policy responses and ground for it to land, and (6) seize the first-mover advantage when the opportune moment comes. This yields the following list of promising projects one could pursue: Developing a theory of warning shots based on existing precedents (e.g. GPT-3.5 and Mythos releases), and past analogies: developing a typology and predicting likelihoods. Building infrastructure to catch a warning shot when it happens. (For example, Ajeya Cotra talks about recurring internal intelligence explosion evals of AI labs.) Creating strategies to avoid gradual numbing / sleepwalking through a warning shot (presumably there are lessons from e.g. communication on climate change). Building preliminary infrastructure (institutes, think-tanks, lobbying parties...) that lays out a good ground, for when a warning shot and its policy response would happen. For example seeding right world models for public and policy makers, so that when warning shot happens they can understand the long-term implications (i.e. AGI dangers). Mapping out good, medium, and bad policy responses based on warning shot types. Building consensus in AI safety community of what policies to advocate for and against jointly. Building infrastructure for lobbying the right parties at the opportune moment. For example: what are the key decision-makers for each warning shot - policy response pair? Which organizations / individuals have the most legitimacy among these politicians and policymakers? Two Important Notes: This is not a call for people to induce a warning shot.[1] AGI governance strategy should not over-rely on warning shots.[2] 1. Understand Warning Shots The AI Safety community currently places a lot of hope on warning shots inducing international cooperation on AGI risks. It would be useful to better understand the dynamics that lead from warning shot to international cooperation. How likely are we to get a warning shot prior to unacceptable risk? How significant would the warning shot have to be, and what other conditions must be met to open a policy window for international cooperation? How do we strike the right balance between attempting to galvanise action after every minor “warning shot” (at the risk of being dismissed for “crying wolf”) and waiting for a major event (at the risk of acting too late)?[3] Warning shots could include alarming safety evals, the release of a strong AI agent (another “ChatGPT moment”), widespread automation of white collar jobs, minor/major accidents, misuse incidents, etc. To make the most of warning shots, it would be useful to characterise different types of warning shots, predict how likely they are to occur, and anticipate what the expected public / policymaker response is likely to be for each type. A useful frame is Kingdon’s three streams model. A warning shot mostly affects the “problem stream”: it makes some latent risk suddenly feel real. But international cooperation on AGI risks will only become plausible if the “policy stream” already contains credible proposals, and the “politics stream” contains enough elite, public, and institutional support. The practical implication is that warning-shot preparation cannot just mean “better messaging after the event.” It requires pre-building policy options, coalitions, legitimacy, and channels to decision-makers. 2. Prepare to Catch Warning Shots Preparing to catch warning shots requires a detection stack: capability evaluations (especially labs entering an intelligence explosion), alignment evaluations, incident reporting, compute and deployment monitoring, whistleblower channels, and more. For certain types of warning shots, we will only get a timely warning if we build such infrastructure beforehand. The AISI network could become the institutional backbone for warning-shot detection. UK AISI was founded on the mission of "minimising surprise" from rapid and unexpected advances in AI, which is almost exactly the institutional version of “catch warning shots early”. 3. Avoid Getting "Frog Boiled" The release of ChatGPT served as a wakeup call because it caught people off-guard. With AI labs releasing new, incrementally more powerful models every week, we risk reaching dangerous capabilities without this resulting in a single, clear warning shot. Similarly, different organisations publishing a steady stream of increasingly disconcerting safety evals may be less impactful than e.g. the network of AISIs publishing a prominent report every half year which summarises the results of all safety evals. Rachit Dub