Policy career planning in the age of imminent superintelligence

Crossposted from my blog. Nearly all career advice rests on an unstated assumption that the world your career operates in will look roughly like the world you trained for. The idea was that if you spend six years studying in a PhD, the field you studied will still be there and still look approximately the same, still hiring and still moving at a pace where your accumulated expertise compounds. For nearly all of human history, this assumption held well enough that nobody needed to state it. I don’t think this assumption works anymore. Now we are entering a phase that may be called the AI “midgame”. Stories about “AI risk” are no longer just future hypotheticals — AIs are now capable enough and misaligned enough to break out of their own companies and coordinate to attack other companies. Discourse around AI is changing very rapidly, where policy ideas being considered this month would’ve been laughed out of the room just four months ago, and with a lot more people interested in engaging than before. And things are only going to get more intense. Progress toward superintelligence — AI systems far more capable than any human at essentially all cognitive work — is currently underway. It seems likely that around four years from today, and potentially just one year from now, we will be in the position of recursive self-improvement — where AIs can fully automate the AI research and development process. At that point, reasonable, well-informed people disagree about what happens next. Perhaps we coordinate to hold off on recursive self-improvement to make sure we know what we are doing. Or perhaps we end up with AI superintelligence and then perhaps we are all dead, or perhaps we are living in a utopia, or perhaps somewhere in-between, or maybe there is still a lot to do and we just have the AIs do it while we relax at the beach, or maybe there still ends up much more for humans to do and we do that. But regardless of which path happens, the world will be a crazy different place, and it will be very difficult to plan a career around it. Instead, careers are best planned around recursive self-improvement and superintelligence, and that this looks very different from typical career planning. If you take AI superintelligence seriously, it thus seems like the next 1 to 4 years will be critical to get right and we collectively have only a few more big bets we can make and we need to make those bets count. Modes of impact In what I’d call normal-mode career planning, impact accrues roughly linearly. In this mode, you earn about 2 “impact points” per year across a 30-plus-year career — some years a bit better, some years a bit worse, and perhaps overall a gentle upward slope as seniority compounds. Your total lifetime impact is then the sum of this slow upward climb. Total lifetime impact is the area under a long, flat-ish curve. Here, the rational strategy is to invest heavily in credentials early, accept low-impact years as the price of capacity-building, and optimize for the long grind. If you’re following normal-mode policymaking, you might do things like identify problems in advance, develop strong policies and legal frameworks before issues fully materialize, and engage in the slow and unglamorous world of legislative drafting, regulatory design, standards development, and coalition-building. But in superintelligence mode career planning, the curve looks nothing like that. Impact is chaotic and lumpy. A handful of critical windows may account for the large majority of your total career impact, and you cannot reliably predict when they’ll open. You can’t count on building a large body of work over a large period of time, because your body of work may obsolete faster than it compounds and you don’t have a large period of time anyways. The strategy that maximizes expected impact under this model is different — prepare broadly, position yourself near where windows open, and maintain the agility and flexibility to act fast when they do. For superintelligence mode career planning, it might be more instructive to think about other imminent crises where we all have to react very quickly and forcefully to prevent it from becoming catastrophic. Think more like March 2020 for COVID — the ~1 year intense period when years of proactive mode pandemic preparedness work (or its absence) suddenly got thrust into the spotlight and everyone rapidly tried to orient to what is going on and what to do about it. If you look at the people who mattered most in March 2020… they were not necessarily the most credentialed epidemiologists. Some had spent years in proactive mode building pandemic playbooks that suddenly became important, but others had no special preparation but were simply positioned well and reacted quickly. My suggestion is that you want your policy work to be a portfolio of proactive mode work now while simultaneously maintaining the relationships and flexibility that let you act decisively when chaos mode arrives. Ask your