What just happened? A retrospective of AI alignment

This sequence is about the last decade in AI alignment. Over five posts, it recounts the gradual transition from a field which treated alignment as a hard scientific problem, to a field which has largely abandoned the goal of deep, generalizable scientific progress in favor of iteratively improving existing systems and attempting to gain technological and political power. I also describe (in subsequent posts, which I'll upload over the next few weeks) how fear and (self-)deceptive reasoning made the field one of the biggest forces pushing AI capabilities forward over the last decade, especially via significant contributions to the scaling of LLMs and the development of ChatGPT. Zooming out further: the two leading AGI companies, which are locked in an intense rivalry, were both explicitly founded under the banner of AI alignment, and got off the ground in significant part due to alignment-oriented ideas, talent and resources. People in the field often sense that something must have gone wrong to get here, but don’t know how to allocate responsibility (aside from blaming Sam Altman and sometimes Elon), and fall back on assuming that “the ship has already sailed”. But in this sequence I characterize our current situation as resulting from a pattern of mistakes which is both recognizable in the past and actively ongoing, and which if continued will cause similar kinds of dysfunction over the next decade. To be clear, I’m not taking a strong stance in this sequence on whether AI will go well or badly—that seems up for grabs. My concern is that the alignment community had a plan to make good outcomes more likely (differentially advancing alignment over capabilities) but has mostly pushed the world in the opposite direction, while some parts of it gained a lot of power by doing so. This is not trustworthy behavior, and should be a big update about how well the community will use its power going forward. In particular, it’s very bad for the world that the community doing the most to steer the future of AI isn’t really trying to distinguish the extent to which its leaders are sincere vs sycophantic vs power-seeking. I care about this significantly more than I care about the object-level effects of accelerating capabilities, because the integrity and rationality of a few key decision-makers will likely shape the coming decades. And yes, there are others with power over AI (like Sam and Elon) who have less integrity in most ways than alignment leaders. However, in my mind the level of adversarial dynamics within the field makes transparency, integrity and accountability more important rather than less: lacking integrity makes you much easier to manipulate, as I recount in the post on Fear and Anticipatory Obedience. Unfortunately, the alignment community is doing very badly at learning from the past decade, or holding anyone accountable. Indeed, it’s pursuing many strategies which seem likely to recapitulate previous mistakes. Four of the most prominent, which I’ll discuss in the final post, are: Trying to convince the US government to take AGI much more seriously. Doing “alignment research” which is very similar to capabilities-maximizing research (especially building automated alignment researchers). Trusting Anthropic too much (in an analogous way to how we trusted OpenAI too much). Trading off clarity in thinking about politics for conformity (in a similar way to how we traded off clarity in thinking about AGI for conformity to the ML ontology). These and other mistakes are reflective of deeper irrationalities. One crucial pattern is what I call “jumping down the slippery slope”: viewing an outcome as so inevitable that it doesn’t matter much if you contribute to it, in a way which leads you to become a significant force pushing the world further and faster towards the "inevitable" outcome. A central example is Sam Altman’s original email to Elon about founding OpenAI: “Been thinking a lot about whether it's possible to stop humanity from developing AI. I think the answer is almost definitely not. If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first.” Many strategies pursued by the alignment community (e.g. the four listed above) showcase this same pattern. You can view it as a result of individuals inappropriately reasoning about their marginal impact despite their actions often having extremely non-marginal effects (in part because so few others were, and are, taking superintelligence seriously). More fundamentally, you can view it as a result of individuals inappropriately reasoning about individual impact rather than thinking about the policies that they’d recommend for the field as a whole (which more sociology-style reasoning about norms and preference cascades, or FDT-style reasoning about entangled decision-making, would have prevented).  However, even given these conceptual errors, people wouldn’t jump down nearly as many slippery slopes if they we