An "Anthropic Principle" for Formulations of AI Alignment

This post is crossposted from my Substack, Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I explore one path toward formulating AI alignment without needing to formalize humans or our values, looking instead at the well-integrated computational power characteristic of agents capable of confronting the alignment problem. One natural formulation of AI alignment, the study of how to be sure our highly capable software acts according to our interests, starts from unambiguous descriptions of human values (whether an AI starts with hardcoded values or learns them). This concept is mostly speculative, as we are far from knowing how to formalize our values precisely enough. My last article considered two reasons why we might want to hesitate to build in a model of humans. In my mind, the more important reason is that, from an engineering perspective, it seems wickedly difficult to formalize human values precisely enough, let alone formalize values of some extrapolation of “better” humans. There are also other objections that are more philosophical, e.g.: if our future AI systems meet Star Trek-style aliens who are like us but blue, is it really proper to ignore the aliens’ interests? Is it safe to assume that some extrapolation to “better” humans necessarily respects blue aliens, given humanity’s dismal record in other first contacts? For the rest of this article, let’s step back to a technical problem: how do we formalize identification of which agents alignment should focus on? We have one conventional “test case” that is almost too obvious to state: in our present circumstances, this rule should identify people as the agents to focus on. There should also be good reason to believe the rule makes proper decisions in other circumstances. We also want it to be plausible that the same rule is substantially easier to implement in software than any approach based on reverse-engineering human values. I’ll start with a big-picture consideration of how intelligence has increased in the universe over time, but I promise I’ll return to the original goal. Who Could Think About Alignment? The anthropic principle explains the apparently improbable fact that we inhabit a universe capable of developing intelligent life. An awful lot of random chance needed to line up properly to produce us. However, the question “how did we get here?” can only be asked by life intelligent enough to think abstractly and formulate the question. Yes, there may be many possible universes compatible with a broad formulation of physics, and likely only a small minority of them are compatible with intelligent life having evolved. Hence, contingent on someone being able to ask the question of how we wound up here, we must be in one of the universes conducive to intelligent life evolving. Let me now develop this kind of thinking toward a principle that any agent at roughly the computational scale to support thinking about AI alignment should be defined as a proper focus for alignment. Instead of making a somewhat arbitrary choice up-front about which agents qualify, let’s try to define alignment once and for all to capture all beneficiaries. (We’ll come shortly to a twist that keeps literal application of this idea from taking us where we expect, motivating a fix.) In this explanation, I’m going to embrace the computational theory of mind shamelessly, because it has always felt natural to me. This theory says that brains are understood as special kinds of computers, so thinking about computation broadly subsumes thinking about (biologically embodied) intelligence. For now, we’ll discuss at an abstract level, but the next section will concretize to societies of humans (not atomized individuals) as critical computational systems. Some computational architectures can host the kind of abstract thought that leads to asking big questions about the universe. Some can’t. Let’s simplify for the moment and think in terms of one linear dimension of computational power. Consider how much such power is needed to be able to design recursively self-improving AI. Presumably there is some relatively well-defined minimum level. If self-improvement is kicked off thoroughly enough to cause an intelligence explosion, then quickly all the rules change, and the familiar alignment problem no longer applies, or at least its context becomes so different that the ideal rules are probably different. For example, AI designing AI is likely to have better explicit, formal/algorithmic understanding of its own values than we do, if it is successful creating its own successors. These observations combine to suggest that there is a narrow window of computational power that naturally evolved intelligence can have when it is working on formulating alignment: it must have passed the minimum power threshold but not yet have begun an intelligence explosion. The window is narrow if we assume that solving