Distillation of the AI2040 Alignment Roadmap

This is a distillation of the AI2040 alignment roadmap (with some parts also drawing from "How do we (more) safely defer to AIs?"). All ideas expressed are by Ryan and Thomas[1]; I only created the diagrams and substantially changed the structure to make it more accessible.[2][3] Note that the original post has this disclaimer: "This supplement is a very rough draft representing in progress thinking. It should be interpreted as internal research notes, not as a final product that we are entirely happy with." Quotes in this post without explicit attribution are from the roadmap; the first word of each quote links to the quoted passage. TLDR: The roadmap outlines a plan for what a responsible frontier AI company in a Plan C scenario should ideally focus on, including a lot of notes on what decisions should change depending on how reality deviates from the modal scenario. This post is already a summary of the roadmap. If you want an even shorter summary, you can check the summary section of the roadmap. This post disentangles the macro plan part from the safety strategy part: Macro plan: Which course the AI company takes through the intelligence explosion (most importantly how long to keep scaling and via what route to exit the period of humans managing everything). Safety strategy: An overview of the current technical plan for how to make the AIs sufficiently aligned and how to get sufficiently good safety work out of them. (These two are of course not independent, and the split isn't crisp. E.g. the pause and handoff decisions strongly depend on the safety state, but I included them in the macro plan because otherwise the safety section would get too full. The macro plan can be seen as the higher-level part of the overall plan.) In case you don't know precisely what a Plan C scenario means, here are the Plan C assumptions. Plan C Assumptions Responsible Leader. We assume that a responsible AI company is in the lead. Their leadership is willing to take very costly actions in order to mitigate AI takeover risk (e.g. slow down for several months). Despite this, we’ll assume that by default, they’ll have a bias towards proceeding and being willing to take on a large chance of takeover risk to increase their chances of winning. 1-12 month lead over their competitors. Either trailing US or international AI companies will overtake the leader within 1-12 months. Though we mostly won’t focus on this scenario, Plan C is also applicable in the case where the USG is sufficiently worried about AI takeover that they are willing to regulate AI companies to mitigate risk as much as possible without losing to China, and without being willing to sabotage China. This plan focuses on making effective use of limited resources to spend on safety and doesn’t ensure a high degree of safety (as the level of resources we’re imagining is insufficient for this). Indeed, arguably the most likely outcome of this plan is misaligned AI takeover. Due to having limited resources, this plan will involve proceeding in some situations where the level of risk is substantial (because exogenous risks are higher) and we won’t necessarily gain much confidence about the exact level of risk. Note that we expect that something much less reasonable than this plan will happen in practice, so while this is a recommendation (for what to do with limited resources), it isn’t a prediction. 1. Macro plan You can see the diagram with some minor additional interactive content here 1.1 What are the 4 phases? 1.1.1 Phase 1: Preparation This phase has already begun and is defined to last until ITAI (Initial Transformative AI), which roughly speaking is the point where 3x AI R&D labor acceleration is reached.[4] During this phase, the AIs are sufficiently weak that they don’t directly pose major existential risks and aren’t transformatively useful for accelerating safety work. 1.1.2 Phase 2: Mitigation + Elicitation This phase starts at ITAI and lasts until the start of the pause in which the lead time is burned for safety. There are 2 important differences from Phase 1 that likely start to pop up around that point: (1) AIs are sufficiently capable that scheming is a serious concern (scheming is both plausible and scheming AIs are capable enough to cause serious problems in the absence of countermeasures) and (2) AIs are very useful for accelerating existentially useful work. 1.1.3 Phase 3: Pause This phase starts when the pause starts and ends when we hand off to AIs. (Note that this phase will likely burn most but not all of the lead time. There needs to be some remaining lead time for the AIs post-handoff to proceed safely.[5]) By default, the pause should start when AIs reach the Min-H capability threshold, defined as follows[6]: Min-H AI (Min-Handoff AI): AIs which have a level of overall general capabilities which would suffice for HTC-handoff being viable. (As in, the capabilities would suffice for our HTC-handoff desiderata if the AIs were also