Alignment & Succession: The Two Bars of Alignment

(Originally published on No Set Gauge June 24th 2026.) Rembrandt, Saul and David When people talk about AI being “aligned”, I think there’s a lot of conflation between two different bars of success: The AI does what you say and does not go rogue. If you ask it to do a machine learning experiment for you, it actually does that instead of scheming against you and escaping onto the internet. If you say “stop”, it stops. (Some AI properties considered important for achieving this bar are corrigibility, non-deceptiveness, and intent alignment.) The AI has fully internalized our values, and could run society, and the lives of the humans in it, in a way that we would judge as good. You just ask it to build the ideal society, and it figures out what utopia is and builds that for you, and it really is utopia. (Some concepts related to this: value alignment of the AI, the AI achieving humanity’s coherent extrapolated volition) The first might sound like a very low bar, and it is. It’s also a standard we’re familiar with from other technologies. With nuclear reactors or airplanes, we ask the question “will it blow up in our faces?”. Just like it’s possible to build a nuclear weapon, it’s possible to build an AI that blows up in our faces very badly. The difference with AI is that there’s a case for it being much more powerful, and our understanding of whether it blows up in our faces is much weaker than that for nukes. With nukes, after all, we have nuclear physics that works very well. With AI, we have long theoretical arguments about what superintelligence would look like, whether it’s possible, and whether it is extremely likely or extremely unlikely to blow up in our faces. The second might sound like a very high bar, and it is. It’s also a bar that we’ve not previously had to think about when talking about technology. If we’ve thought about it, it’s been in the context of arguing about culture, politics, or institutions. Now you might say: what do you mean the AI is meant to run our entire society and all of our lives? Isn’t that a ridiculous strawman of what people want AI to do? And perhaps it would be good if it was, but it’s a common view among people who expect AGI soon (and especially, perhaps, the AI safety community). On LessWrong, my posts have previously encountered opposition to the idea that it’s worth doing any planning or design of the post-singularity society. Why would we, with our weak fallible human minds, waste our time designing paradise, when soon we will have a god—sorry, I mean a superintelligence—that does it for us? A common view I’ve heard is that all that matters is that we handle the transition period, and build aligned superintelligence, and then we delegate all of our decisions to it. Will the superintelligence implement democracy? Will it rely on markets? These are just earthly institutions, the view goes, that we humans have built as kludges. Probably the superintelligence will transcend all such things. Does this mean a tyranny where the superintelligence micromanages everyone’s lives? Well, it’s superintelligent. If that is good for us, that’s what it’ll do. But to the extent that we value choice, it will give us choice. So no need to worry about tyranny! What is the nature of the good life? The superintelligence will solve this question. After all, it’s superintelligent! Probably—some think—this looks like the superintelligence discovering some optimum configuration of matter, and then tiling the universe with that. But if that’s not the case, don’t worry, the superintelligence will have figured it out and it will just go out and do the right thing. If you want to know the nature of the good life, you just have to ask it, and it will tell you. In his review of Peter Singer’s commentary on Marx, Scott Alexander writes: [...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out. I think this attitude is ridiculous, for reasons I will soon get into. In particular, what’s weird is that the AI not blowing up in our faces is treated as approximately of the same difficulty as the AI knowing everything about what is good and right. But if you look at where it came from, this makes sense—given a very particular set of assumptions. Like much else in AGI discourse, it all comes back to Eliezer Yudkowsky. Yudkowsky’s central case for doom comes from a combination of two points: The fragility of value. It’s hard to pin down what human value is. Any simple metric, maximized