The AI Race is Not a Prisoner's Dilemma

Statement on AI use: AI models (mostly Fable 5 and Opus 5) were extremely helpful in 1. iterating through lots of variations on the game theory models presented, 2. helping to confirm my understanding of the math, 3. fact checking, finding sources, and catching errors, and 4. providing editorial feedback. The game theory visualizations are entirely vibecoded; I have checked the math, but not the code. All prose is my own except where marked and I take full responsibility for everything in the post. I. Introduction When I teach my philosophy students about existential risk from artificial superintelligence, many students' first reaction after fully grasping the argument is "Why are we still building this?" And the general public arguably shares the sentiment: an Opus 5 agent found that 63% of Americans told Pew in February 2026 that AI is advancing too quickly (against 2% who said too slowly), and a Yahoo News/YouGov poll the previous October found that 53% think it at least somewhat likely that AI will 'destroy humanity' someday. Why not just shut it all down until we know it's safe? The Discourse has an answer to this question: it's because AGI is an arms race. Here is Leopold Aschenbrenner in 2024: Some hope for some sort of international treaty on safety. This seems fanciful to me. The world where both the CCP and USG are AGI-pilled enough to take safety risk seriously is also the world in which both realize that international economic and military predominance is at stake, that being months behind on AGI could mean being permanently left behind. If the race is tight, any arms control equilibrium, at least in the early phase around superintelligence, seems extremely unstable. In short, "breakout" is too easy: the incentive (and the fear that others will act on this incentive) to race ahead with an intelligence explosion, to reach superintelligence and the decisive advantage, too great. At the very least, the odds we get something good-enough here seem slim. (How have those climate treaties gone? That seems like a dramatically easier problem compared to this.) (Source) And here is Hunter Ash earlier this year: AGI is not analogous to nukes I should start by saying I'm about as far from an AI safetyist as it's possible to be. But I'm going to engage with that frame, and show why the situation is in no way analogous to that of nuclear weapons, which is their only real example of the kind of international coordination that would be required to stop AI progress. The basic issue is this: you can build nuclear weapons but not use them. And this is what mutually assured destruction hangs on. On the safetyist view, you cannot do this with AGI. Making it and "firing" it are the same action. The game with nukes is this: if we shoot nukes at another nuclear-armed country, there is an extremely high chance that every major population center in our country gets wiped out by their nukes. And the same is true for them. So no one fires. No one defects. With AGI, the only way to not defect is to not be able to return fire if the other side defects. Unless your P(doom) is well north of 50%, then the deterrence math doesn't work. If the other team builds it in secret then turns it on, and they did achieve decent alignment, then you're absolutely screwed. There's no mutually assured destruction. They win and you lose. You can't suddenly build your own in retaliation when the missiles are already in the air, so to speak. I also think compliance with any treaty would be near-impossible to enforce, but that's a subject for another post. We cannot get out of this arms race even if we wanted to. Winner takes all, and there is no pause button. (Source) So the argument is basically that if we don't build it, then they will, where "they" could be DeepMind or Anthropic or OpenAI or China—especially China. And if they build it, that would be worse. We'd like to stop, you see, but we can't. That is to say, it has the structure of a Prisoner's Dilemma. A Prisoner's Dilemma is defined by the fact that Defect is the dominant strategy, even though both players prefer (Cooperate, Cooperate) to (Defect, Defect). The argument is that, even though both the US and China (or both Anthropic and OpenAI) would prefer that they both not build, they also both prefer that they themselves build, regardless of what the other does. The problem is that the AI race is not a Prisoner's Dilemma. I'm not the first to point this out—Katja Grace has talked about it in several posts over the years, and there are a couple of papers making a similar point.[1](And Zvi actually made the point in one of his newsletters while I was in the process of writing this post.) But since the Discourse at large still seems to mostly take the "AI is a prisoner's dilemma" meme for granted, I want to go through in detail why it is not. II. Nuclear Arms Races A classic "arms race" (e.g. nuclear weapons) is a Prisoner's Dilemma. A Prisoner's Dilemma is defined by four payoffs