The Scramble: getting in position to pace the frontier

Crossposted from my Substack. ~ Suppose the President summons the AI CEOs and his top national security advisors to an emergency meeting at the White House. He has become extremely concerned about superintelligence — the possibility that AIs far smarter than humanity combined slip beyond our ability to correct or shut down. If that happens, there is no way back. The President is concerned humanity could become permanently out of the driver's seat of its own future. He wants to figure out what to do. The reaction is panic, chaos, confusion. The President asks questions. The AI companies are blazing toward superintelligence at high speed — can we slow down as we approach the dangerous thresholds? …Some of the AI companies say they don’t have a good plan to slow down or stop, especially as their competitors may just undercut them if they do. What’s that about? What's going on with China — can we get them to pace as well? Can we get a deal without Beijing sneakily catching up and maybe surpassing us? And if there's no deal to be had, what then? More like the Cuban Missile Crisis than the NPT I sometimes hear people talking about treaties and other kinds of extensive international agreements as the way we would manage advanced AI. But treaties typically take many years to operationalize. You want to be concrete and clear about your definitions. You want hundreds of technical experts on both sides informing nuanced diplomatic discussions about various details — numbers of missiles, types of missiles, timelines for disarmament, and so on. The Nuclear Nonproliferation Treaty took three years of negotiation and two more years to implement. However, I expect that the moment the President is getting serious about superintelligence won’t feel like an extended treaty negotiation. It will feel less like the Nuclear Nonproliferation Treaty and much more like the Cuban Missile Crisis. The Cuban Missile Crisis lasted thirteen days. The vibes were insanely tense, stressful, chaotic, and confusing. The key decisions were limited to a group of roughly fifteen individuals — the Executive Committee of the National Security Council, or “ExComm” — with President John F. Kennedy himself spending a great deal of time shaping and steering discussions. There was incomplete information, large uncertainty over the intentions of the Soviets, warring factions within the US and Soviet bureaucracies attempting to sway senior decision-makers, and a lot of chaos. I expect the initial phase of AI superintelligence management to share these features. There will not be a multiyear process to set up a technical bureaucracy that understands advanced AI risk or compute verification proposals. Instead, we move forward with what we have. The President has to quickly make critical decisions that may lock us into particular paths. We rapidly develop a national strategy for what the US government does about recursive self-improvement (RSI), frontier model security, compute verification, technical evals and safeguards, and China. A scramble and then three phases How would we go from this Presidential emergency meeting to a deal with China to pace the frontier? My rough sketch is it could proceed with a scramble and then three phases[1]: The scramble (roughly 2-4 weeks): The government has definitively decided they want to take strong action on superintelligence and has to get an initial deal done. Phase 1 (buying ~3 months): the interim deal. We make an interim deal to have time to get to a better deal. This may involve strict requirements on data centers above a certain threshold to delay the training of an AI model that could do recursive self-improvement. This likely relies primarily on executive action. Here, monitoring and verification rely on traditional methods, and nations are willing to agree to scrappy, janky, and invasive things — but we have a whole-of-government effort to build the tools for something more durable. Phase 2: the durable deal. Phase 1 (interim deal) is meant to get us to Phase 2, where we have a stable deal between all nations where uncontrollable AI is reliably prevented. Congress has potentially been involved by this point, other countries are brought in, and the verification program matures into something that looks and feels more like a treaty than a haphazard scramble. Phase 3: safe superintelligence. If we want it — once we can figure out how to do it safely and have sufficient buy-in. The near-term intellectual work is unevenly distributed across this structure. Significant detail is ironed out during the scramble and during Phase 1. The scramble: What questions does the President ask? The President launches an emergency meeting to figure out what to do about advanced AI, recursive self-improvement, and the road to superintelligence. Some questions that are likely on his mind: How dangerous is it to proceed through recursive self-improvement and superintelligence without pacing? How dangerous is it to proceed if w