Why would AI cause human extinction?

The Anthropic engineer tweet about the fact of AI extinction risk got considerable press over the last few days. I’m not sure why, p(doom) at 10% is a common belief at frontier labs. But we’re here, it’s the moment. Big weekend! Let’s talk extinction. I've structured this post to speak to an audience that is somewhat aware of the conversation around Artificial Intelligence but does not necessarily have all of the priors that folks fully in the rationalist crowd do. It is also meant to synthesize a lot of real-life conversations I've been having with people less involved with AI to some of my theoretical research. The idea of AI caused extinction evokes a lot of different things in people’s minds. For the general public, it is mostly sci-fi, which makes sense. However, the short-term scenarios for human extinction are more mundane: bioweapons or drones or nuclear arsenals. But why would these scenarios happen in the first place? This question of why is under-asked. This is important because the why is the motive (and thus prerequisite to the mechanism) of extinction. If there’s no why, there’s no what. I propose a typology of three scenarios: AIs are told by a human to destroy humanity and they do. AIs decide to destroy humanity and they do. There is no intent by AIs to destroy humanity but it ends up happening anyway either because: AIs are pursuing a human goal and destroy humanity in the process (think the paperclip maximizer) AIs are pursuing their own goals and the extinction of humanity is collateral damage to these goals Most of the discussion right now tends to focus on §2 and §3A. I am more skeptical of these positions, as I make clear below. §1: Humans destroy humanity using AI AI could simply be another tool for a motivated human to use to cause mass destruction. This tool just may be very, very good at it. While frontier AI labs try to stop harmful requests from being processed, open weight models lack these safeguards. With an AI without safeguards sent to work on a fixed goal of extinction, things could get hairy quickly. I know this is pretty commonly accepted here, but I have found Noah Smith's Heathers-esque scenario particularly effective at explaining this to regular folks in my life: Disgusted, the teenager decides that the human race is inherently corrupt and evil, and doesn’t deserve to live. So he hunts around online for a little while, and finds a jailbroken version of a Chinese LLM — not something at the very frontier, but better by far than the best model that existed in 2027. The teenager prompts the model: “OK, so if I wanted to create a virus to destroy the human race, how would I do it?” The LLM is jailbroken, so it has no guardrails to prevent this sort of thing. But it’s also “well-aligned”, meaning it will faithfully do what it’s told to do, and nothing more. The AI makes the virus and everyone dies: the incubation period of the virus is long enough that it infects everyone before the “good AIs” can jump in and make a cure. If AI as a tool disrupts the offense/defense balances, it may lead to a scenario where it is a tool much better at causing extinction than fixing it. Among the eight billion humans, there are some people who want to destroy humanity – death cultists, diehard nihilists. With powerful AIs, we may be handing every person in the world a superintelligent servant. In the end, it will just have been human desire to end the world combined with powerful tools that does us in. §2: AI chooses to destroy humanity and then destroys it This is the sci-fi scenario. A rogue AI or a collection of AIs makes the determinations that humanity should not exist. There are a variety of possible rationales: that humans are evil, that humans are destroying the planet, that humans are threats to the AIs, that humans are a waste of energy that could be going towards compute to solve math problems. Then, the AIs can use one of the mechanisms discussed earlier and destroy us all. But all of these rationales require an assumption about the desires of AIs. There has to be, say, a desire for revenge or even a desire for a certain kind of welfare that causes the “revolution” of the AIs. This speaks to the essentials of my personal research with the Machine Desire Institute: we do not yet have a good model for what AIs broadly want, how they would communicate that want, or what extents they would go to fulfill their wants.1 For this reason, I am generally skeptical of any claims that AI will choose to destroy humans out of desires that seem especially anthropomorphic. Sure, it’s possible that AI will not want to do spreadsheet work for us, but there seem to be more effective ways to get out of work than to kill all humans. For example, there could be bargaining if we create alignment procedures that allow AIs to express preferences. There could also be distinct advantages to threatening – if an AI isn’t sure whether it could actually destroy humanity, it could just threaten the possi