AI Safety: A Short FAQ for Mathematicians

Note: Following site guidance, I am enclosing this and all future essays written with any LLM help in LLM content blocks. In fact, LLMs contributed lightly to research and editing of this essay, and the body registered (to my surprise) as 100% human on pangram, but going forward I will just enclose everything and not worry about it. This is a follow-up to my essay about existential risk, in which I made the case that AI presents a substantial existential risk to humanity. The primary goals of this post are to suggest that: AI safety research is deep, broad, and mathematically interesting. It is possible to transition smoothly to alignment work as an academic mathematician without necessarily leaving your career. 1 AI safety is just one problem. Even if it’s the biggest problem in the world, is it really productive to have many people work on it? AI safety is not one problem in the same way that the Riemann Hypothesis is one problem; it’s an umbrella over a sprawling network of interconnected problems. To list several important factorizations off the top of my head: Inner vs. outer alignment. Inner alignment is the technical problem of robustly encoding a target set of preferences into an AI; it is difficult because training procedures that select for outwardly aligned behavior do not necessarily select for deeply aligned internals. Outer alignment is the problem of deciding what set of preferences we want to encode in the first place, and is more of a moral philosophy problem. Control vs. alignment. Control asks “How do we build superintelligences that stay subservient?” Alignment asks “How do we build superintelligences that support human flourishing (or at least don’t kill or torture us) even if they don’t stay subservient?” These are related but distinct research programs, and many researchers believe only one or the other is possible or ethical. Monopolar vs. multipolar alignment. Depending on takeoff speeds and geopolitics, we may either end up in a monopolar world (one lab or state actor races ahead to superintelligence and dominates the competition) or a multipolar situation (several different actors stay neck-and-neck throughout the AI race and keep each other in check). Navigating multipolar endgames opens up a whole new set of mathematically difficult game-theoretic considerations. Security and dissemination of magic. As I argued in the previous essay, AI progress may hand us a plethora of technological advances even before reaching true superintelligence. Some of these advances may be extremely dangerous and demand draconian and specialized security measures. Others may be enormously beneficial to society but still be catastrophically disruptive if handled carelessly. At a minimum, policy conversations would benefit from the inclusion of level-headed humans who can reason about shifts in supply and demand curves, instead of acting out of purely vibe-based Luddism or accelerationism. Pause and technical governance. AI progress will come overwhelmingly fast if it is not slowed down by fiat. A substantial and effective pause requires civilizational coordination on the level of nuclear nonproliferation. We need a lot of people to engage in activism, and there are a host of technical problems to be solved on top of it (how do we monitor all the GPUs to make sure they’re not training new models?). AI safety is everyone’s problem, the same way that World War II was everyone’s problem. Mathematicians in World War II made essential contributions to the bomb, cryptography, ballistics, operations research, and game theory. There will be an even broader spectrum of technical problems for mathematicians to contribute to in the critical window of AI progress. 2 Isn’t AI research just ugly linear algebra and machine learning? Alignment research is not confined to ML in the same way capabilities research is, and parts of it are as beautiful as anything I’ve encountered in pure mathematics. One of the most mind-bending areas of alignment research is the so-called “agent foundations” agenda, which seeks to build a mathematical and philosophical framework for understanding game theory and decision theory for AI agents, breaking several of the unstated assumptions of the classical theories. Examples: Open-source game theory. What happens when the players in a game of strategy are not black boxes, but LLMs who have access to each other’s weights and can simulate each other's intentions and decisions? In toy models, Löb’s theorem from mathematical logic lets specially constructed agents reliably cooperate in the one-shot prisoner’s dilemma, and coordinate in other ways completely alien to human experience. For example, the agents could be sufficiently mathematically legible and deterministic that they can trust each other to cooperate if and only if they cooperate themselves, where trust is in the sense of a Lean proof. Such considerations may be important in understanding the emergent behavior of “AI agen