Five counterintuitive insights from Plan A

Plan A contains many things that would’ve surprised me if you had told me about them one year ago. Some of these include proposals that sound wrong or even actively bad on the surface. In this post I defend 5 core takeaways that I think I would have found most interesting if I could go back in time and explain them to myself before we had started writing. Summary:  Relevant ‘safety effort’ and ‘payable safety tax’ are more important slowdown goals than pure slowdown time. Increased transparency helps with some of the most difficult problems during a slowdown: making nuanced safety regulations and reducing power concentration.  Scaling compute is good in coordinated scenarios, but there should be preemptive measures to make compute arms control easier in the event of deal breakdown (mutually assured compute destruction). A slowdown still has major risks including (1) deal breakdown risk and (2) covert project risk. Out of these, deal breakdown risk seems bigger and more underrated.  There's a tradeoff between security and transparency, but if we try hard we can get a pretty great compromise with large amounts of both security and transparency. (1) Relevant ‘safety effort’ and ‘payable safety tax’ are more important slowdown goals than pure slowdown time. Q: When it comes to slowing down, the longer the better, right? Buying 10 years before the world builds crazy superintelligent AIs is clearly better than buying 5 years, right? A: Wrong. Risk will depend hugely on what happens during the slowdown, and pure slowdown time will be quite an imperfect proxy.  The main point of the slowdown is to reduce the risk of AI takeover from building crazy superintelligence,[1] but how do you actually make the most amount of progress towards this? Slowdown time is a pretty good proxy, but there are a few key factors that I think more directly track how much you can reduce takeover risk:  Getting as much uplift as possible from AIs on safety and alignment R&D. To the extent that you can safely scale to more capable AIs earlier, and then elicit useful AI safety and alignment labor out of them, this can drastically increase your total effort on safety and alignment. It might require 100s to 1000s of years' worth of human-speed progress to sufficiently solve the science of alignment to avoid the worst loss-of-control outcomes. With sufficiently smart, fast and numerous AI agents helping with this science though, it might be possible to get there drastically faster.  Spending this safety effort studying the relevant AI paradigms, or AIs in the most relevant capability regime(s). There might be relatively low transfer between different phases of AI capabilities. Imagine doing a long slowdown before the transformer architecture was invented to study safety, just to mostly throw it out of the window because you couldn’t apply your findings to the new transformer-based paradigm. Having the affordance to pay large efficiency penalties for better safety properties. The classic example here is neuralese vs. chain of thought. It is currently very useful for safety/control to be able to do chain-of-thought monitoring. Transitions to more latent-space reasoning (neuralese) might have significant safety downsides. If you are in a verified slowdown, you can mutually agree to pay safety taxes, by banning the unsafe algorithm (neuralese) and compensating with more compute to reach the same capability without it (safety tax). There might be many other tradeoffs like this to make during takeoff, which could drastically affect the risk level. If you do a 5-year slowdown but you do extremely well in these areas (e.g., you pay a large safety tax, scaled safely but fast early on, and then elicited AIs well on the relevant paradigm(s) that had good transfer to future paradigms) then you might have reduced takeover risk drastically more than a 10-year slowdown where you paused at a low capability level, didn’t pay any safety taxes and didn’t spend much of this time in the relevant paradigm or capability regime.  The counterintuitive takeaway here is that slowdowns that are too early or poorly governed can be net negative. An ineffective pause that didn’t reduce takeover risk can put you in a worse position than when you started. For example, you might have burnt your political will for a pause, burnt lead time over other actors, or there might now be more compute (dry tinder) in the world causing a faster, more dangerous takeoff later. I do still think that probably a simple, naive slowdown is still better than nothing, but it's not robustly good. Of course, a well-managed slowdown might also not be robustly good (e.g., in a case where you don’t deal with the dry tinder problem), but I personally think the difference between a well-managed slowdown and a naive one is bigger than the difference between a naive one and no slowdown at all. My estimates of P(great future) in three regimes:  A 10-year well-executed version of Plan A following the pat