Alignment & Succession: Morality Lives in the Human Individual

(Originally posted on No Set Gauge on 2026-09-05) Woman Holding a Balance, Vermeer In this post I present a sketch of a grounding for morality that is human and active. It can be read standalone, or as part of a four-part series discussing the proper relation of succession—the handing away of power—to how humanity should deal with the coming of superintelligent AI. It is necessarily a sketch rather than a rigorous proof of every last point, intended to orient toward some important and often-neglected moral dimensions. Successionism, as defined in the first part, is the ideology that says that a fundamental transfer of power (and maybe even experience itself) from humans to AIs is the right way to deal with superintelligent AI. The strongest of their arguments is that it may be very hard to avoid. But as anyone not posturing for a bit knows, “is” doesn’t make an “ought” and might doesn’t make right. So are the successionists right, morally speaking? I regret to inform you that answering this requires metaethics. For successionism to be right, it must be possible to confidently divorce moral value from humans. However, human felt experience is the only sure window we have into the territory of which moral theories are the map. The meaningful transfer of that felt experience seems hard to do and be sure of, which is a reason for a strong precautionary principle. Not only are humans in the abstract indispensable to human morality but so is actively weighing options on your own and making choices and undergoing value change. This, and ghosts of history we should heed, argue that the active participation of the human individual is critical to our morality too. Morality comes from humans The fragility and slipperiness of moral value discussed previously often leads to one of two responses. Sometimes the response is a mental panic that forces a retreat to the familiar solid terrain of equations and objectivity and abstract principles. I think this is behind answers like “the ultimate good is complexity” or “the fundamental value is intelligence”, or neo-Pythagoreanism. The other response is that it’s too subjective: sorry, we just can’t say anything about values, anything goes and all is arbitrary. I claim: much like your eyes give you access to evidence about the physical world we live in, your felt moral intuitions give you access to a moral world that your sense of “ought” is unavoidably based on. The inescapability of moral intuition Humans clearly have moral intuitions: you think that being in a nice forest, or being in love, or making someone smile are good, and you think being sick or being treated unfairly are bad. However, it might feel reductive to argue that this is what morality is grounded in. There are two ways to disagree: There is something “higher” that instead defines morality. Most prominently this is the religious view. As I am already trying to derive the correct metaethical grounding of moral theories in one blog post, to keep scope reasonable I have elected to not also settle the question of God’s existence right here. I think the arguments below work without a God, but if you think that God is a faster path to similar conclusions, or explains why we have access to these moral intuitions, go ahead. In these dark twilight years after Nietzsche delivered God’s eulogy or whatever, there are fundamentally two remaining types of things you can appeal to: properties of the universe (”facts”) and properties of your experience (”feelings”, “felt experience”, “intuition”, etc.). The most pernicious form of the first type is some form of might-makes-right, such as the idea that evolution or thermodynamics necessarily point toward what is right. The best argument for this is that natural selection, and if not natural selection then at least thermodynamics, does in fact eventually win. This of course is ridiculous as an argument about an “ought”: if tomorrow we learned that entropy were reversible, or Moloch were slain forever and natural selection replaced with some artificial selection, there would be no change to the definition of right or wrong. The impulse to identify your morals with the winning physical principle comes from a combination of two factors: first, the laziness of wanting to win definitionally rather than on merit, and second (in the West), from a lingering neo-paganized Christian identification of the set-in-stone trajectory of the universe being one and the same with the source of its morality. Once you’ve ruled out the “facts”—the idea that somewhere in the equations of electromagnetism, or written into the sands of Mars, or otherwise out in the physical world, there is the answer to what you should consider right—what you’re left with as a basis for morality is that it must relate to something about your felt moral intuitions, or at least something that lives in your experience of the world rather than the external world itself. You can argue about what aspect of the human ment