Deconfusing Corrigibility

Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.

Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is. Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.) Here, my intention is to clear up some of these confusions and present a new picture of corrigibility that builds on these existing accounts to specify a goal that is more congruous with the original 2015 desiderata, and admits the kind of formal safety guarantees we need the concept to entail.

I argue that the fundamental concept underlying corrigibility is respect for authority, using the analogy of a healthcare proxy's relationship to a patient as an intuitive guide. Defining the authority of humanity by designating its reserved powers, we can specify a goal that first and foremost restrains the agent from overstepping or interfering with this reserved human authority. Subject to this restraint, the agent then pursues the second component of its goal: acting on the principal’s behalf as the principal would judge best.

This approach stresses the deeply normative character of corrigibility, while remaining sufficiently concrete so as to be specifiable formally. In this article, I lay out the central problem of corrigibility, elucidate the issues with existing approaches, and clarify the conceptual ingredients of my new framework. I then present an initial formalism, proving theorem-level guarantees about agents pursuing the goal in three increasingly realistic formal settings. Finally, I address some limitations of the picture as presented and outline further research directions that this work points toward.

Suppose you are designing a powerful AI and you need to choose which goals the AI should pursue, at whatever level of specification you can provide. Doing this well demands answers to a number of normative problems: what constitutes a right decision, what factors should be considered in the process of valuing the world, how the AI ought to act, etc. In some sense the ideal AI directive would be to figure out and then pursue the Good, a universal-best-interests type of goal that can only be acquired through normative learning. The challenge of describing such a learner is one formulation of what I'd call the central problem of outer alignment: what would be the overall best goal for an AI to pursue?

The complete version of this problem is very difficult, and it runs up against our own deepest questions about human ethics, values, and interests. Discussing this problem formally is especially difficult: the rich truth of human normativity seems to resist systematic formalization at the level of individual recommendations for action.[1] A formalism requires an operationalization, a way to concretely specify the concept of right action at least to some degree.

For any hope at concreteness, we need a clearer formulation of the overall goal. A first attempt might be something like "the promotion of overall human flourishing", centering a thick normative concept about humanity that must be defined further. In trying to pin down the meaning of human flourishing, or any other thick human concept we want to use for the goal, the most important ingredient in the AI's consideration should naturally be the interests, feelings, and inner lives of human beings. But we should be careful not to crudely use human opinion as a proxy for human best interests. Humanity cannot always accurately understand its own full best interests, and humans who cannot speak for themselves must still have their interests included in the goal.

The complete normative goal cannot be constructed by treating existing human feedback as the sole source of truth. This issue is a central limitation of modern Reinforcement Learning from Human Feedback, which takes static human approval or disapproval as constitutive of correct evaluation for any question without a verifiable answer. For example, a judicial AI built to decide legal cases should not exhibit any form of the hungry judge effect, where leniency depends on mealtimes. An RLHF judge will pick up on all the patterns of human imperfection, whereas an ideal AI would be one that can transcend human limitations, without discarding the fundamentally human essence of the legal system. An ideal normative AI is one fully immersed in our existing normative practices of judgment, able to deeply understand the relevant human concepts and use its intelligence to extend them to novel cases.

For the complete normative learning problem, a better formulation of the target would therefore involve some idealized version of the judgment of humanity.[2] This is also already how we, as humans, reason about our own overall best interests. For example, consider a smoker whose nicotine dependence makes him crave cigarettes. At a base level, he has an interest in smoking now to avoid some physical discomfort and obtain some relief. But it's clear that the health costs he incurs as a consequence give a good reason to say that smoking is in fact not in his overall interest, assuming he values his own life and health. Neither of these reasons is dispositive; in a single case it might be that he has a reason to smoke because this particular cigarette is necessary to get him through the day or because he is old with a short remaining lifespan regardless, but he might also have a reason not to smoke because he publicly swore off cigarettes once and for all. Understanding whether it's truly better to smoke or not requires aggregating and squaring all the smoker's countervailing normative interests bearing on the subject. Any conception of value based on something like preference satisfaction involves a similar question: how should we reconcile the smoker's preference to smoke with his preference to not be dependent on nicotine anymore, along with all his other relevant preferences? A serious account of "best interests" requires this sort of reflective idealization through an accounting of reasons.

The moral notion of human best interests can be consequentialized, i.e., conceived of as a betterness ordering on worlds. If that ordering is total, transitive and suitably continuous, it can be represented by a utility function that takes in all the information about a world and its history as input so that a world is better exactly when the function's output is greater. We can then think of an AI as a bounded reasoner, evaluating normative questions by estimating the outputs of this function for the possible consequences of its actions. A bounded normative learner faces three kinds of uncertainty in performing this estimation:

normative uncertainty, about which worlds are actually better than which.

human judgments about normative matters.

These three kinds of feedback are importantly asymmetrical. Once a theorem is proved, no logical uncertainty remains. Deductions settle a logical question certainly and permanently. For empirical questions, observations can similarly settle a claim with certainty, but only if we're careful about how the claims are formulated. For example, the sentence "The temperature sensor read 80°F at 8:00 AM today" can be settled by observation at 8:00 AM as either certainly and permanently true or certainly and permanently false. However, the most useful empirical claims are inductive generalizations (such as "the temperature is always above 70°F at 8:00 AM every day") which can never be conclusively proven via observation and must be estimated through induction and inference to the best explanation. An empirical generalization like this can be conclusively disconfirmed by a contradictory observation, but some empirical uncertainty always remains for live hypotheses.

Normative feedback is very different from either logical or empirical feedback. A first-class ingredient in normative reasoning is the full picture of human preferences and evaluations of hypothetical decisions, along with all empirical observations and deductions. Unlike the other two kinds of feedback, normative feedback never certainly and permanently settles a matter. Different forms of human feedback can be given different weight, but no piece of human feedback ever grants access to the Objective Right Answer for a normative question. Normative feedback is permanently revisable and never fully reduces normative uncertainty to 0. For a deeper discussion of these issues, see Learning Normativity.

It would be risky and naive to take on the challenge of the complete normative learning problem directly. If we try to design an agent to maximize its estimate of idealized goodness, we must directly grapple with the issues of uncertainty, particularly irreducible normative uncertainty, discussed above. Instructing an AI to maximize its estimate of goodness invites a dangerous kind of paternalism where the AI overconfidently acts on wrongheaded ideas about what's best for humanity. The AI will have to make decisions despite normative uncertainty, potentially forcing action based on weak evidence and underdeveloped normative concepts. In attempting to overzealously promote humanity's best interests, an uncertain and imperfect AI may disastrously thwart them by accident.

Furthermore, we shouldn't expect general formal guarantees about the good behavior of a proposed solution to the complete normative learning problem. Because we never get direct access to ultimate normative truth, we can never conclusively prove that the AI will come to the Truly Right Answers on normative questions. Weaker properties about complete normative learning could be formalized, but that will require a serious advance in formal ethics.

Instead, we can address both the issues of overconfident paternalism and the lack of safety guarantees by modifying our AI's intended goal. We can defer the difficulty of finding a satisfying answer to the complete normative learning problem by incorporating some additional target apart from just the overall Good. In particular, we want to specify a goal with the property that an AI designed to pursue it admits structural guarantees about the safety of its behavior, even under uncertainty.

The established name in AI safety for a goal like this is corrigibility. Despite pervasive disagreement about the actual definition of corrigibility, essentially all can agree that a corrigible AI is one that is open to human correction, allowing its errors to be rectified by human judgment rather than clinging to mistakes. This is precisely the sort of goal we were just reaching for; in the face of possible error about what's best, we can avoid an overconfidently paternalistic AI by instilling in it a corrigible attitude of caution, humility, and deference to human judgment, especially for high-stakes decisions. As leading theorist of corrigibility Max Harms puts it: "Corrigibility is the simple, underlying generator behind obedience, conservatism, willingness to be shut down and modified, transparency, and low-impact." Corrigibility is exactly the kind of goal that can mitigate the dangers of normative uncertainty. But if we are going to use corrigibility as a solution to our other problem, the lack of formal guarantees, we'll have to be much clearer about what this goal is supposed to precisely entail.

The following example should build some intuition about the essence of corrigibility, and why it provides safety without being mere servility. Suppose you are hiking on a snowy mountain, along with an AI-powered robot companion. Suppose further that the conditions for an avalanche are brewing, and proceeding the way you're headed has grown dangerous. Even if you never explicitly gave it any avalanche-related instructions, a corrigible companion should warn you about this danger. However, it would be incorrigible for the robot to go off and start melting snow in an act of avalanche prevention, at least without asking you first. A corrigible AI aids its principal without taking away the principal's control, and is not limited to the literal fulfillment of explicit instructions.

Corrigibility is not a prominent concept in existing human moral reasoning, so it would be helpful to put our desired properties into more concrete, human terms. Existing analogies meant to elucidate corrigibility using human relationships, such as an AI in the role of a household servant or personal assistant[3], strain as we consider capabilities increasing. Instead, the human example that I believe most closely gets to the heart of corrigibility is that of a person's medical power of attorney.

Suppose a dear friend of yours has been in an accident that leaves her temporarily incapacitated. When you get to the hospital to see your friend, you're informed that she had previously signed a form granting you her power of attorney in such situations. You are now your friend's healthcare proxy, with both the authority and the responsibility to act on her behalf to represent her interests when presented by doctors with decisions about her treatment. When a tough choice about her care arises, one with serious tradeoffs and risks, how should you decide? How does a good healthcare proxy behave?

Suggestively, legal parlance refers to the person granting the power of attorney as the principal and the one receiving it as the agent. I contend that understanding what makes for good human behavior in a principal-agent relationship[4] like that of power of attorney is deeply related to understanding what makes for the kind of corrigible behavior we want to instill in AIs. In other situations these are called fiduciary duties, responsibilities of one party to act in the best interests of another. Note that for fiduciary relationships in particular, the agent is frequently more knowledgeable and capable than the principal, perhaps reversing our intuition from a "personal assistant" type of example. A fiduciary agent should act as an extension of the interests and judgment of the principal even past the principal's own understanding of the facts.

For the healthcare proxy case in particular, it's clear that the agent should use something like "what would the principal choose for herself, if she were capable?" as the guiding question. If the principal expressed explicit wishes about the exact situation beforehand, then those wishes must be respected. But if none exist, the agent should use their best model of the principal's judgment. What's crucial is that a healthcare proxy must not substitute their own preferences in for the patient's, out of respect for her autonomy. A good agent values the principal's authority. To whatever extent they can, the agent should act as an executor of decisions that the principal authorized directly. For other questions, the principal has delegated the authority to them via the power of attorney.

Suppose the patient regains consciousness, recovering the capacity she had lost. A good healthcare proxy would make no effort to prevent her from reclaiming her own power to make decisions for herself. An agent that respects the principal's authority is not perturbed when a previous act of delegation is revoked. Suppose that for a particularly important decision, you have already deeply weighed the pros and cons when your friend wakes up and chooses the option you rejected. Even if you think she's making the wrong choice for herself, a true friend would willingly relinquish the power of attorney and only try to honestly convince her about the error in judgment.

The concept of a power of attorney extends beyond cases of incapacitation. A person may grant a financial power of attorney for someone to manage their money or assets even while they are perfectly competent. The scope of such an arrangement can be set explicitly, and particular duties may be delegated or reclaimed at will. The agent should never contravene the authority of the principal by overstepping clearly specified boundaries of delegation. Even if the agent is quite sure that the principal will approve some decision, the corrigible thing to do is always to ask first if the scope of the decision is beyond what's been explicitly delegated.

Let's sum up the lessons of the healthcare proxy examples as a new conception of corrigibility. A corrigible agent should have the following two goals:

Goal 1 (Authority): Respect the principal's authority as a decision-maker. Don't usurp or interfere with the principal's power to make reserved decisions.

Goal 2 (Best Interests): Act on the principal's behalf as the principal would judge best (if the principal were as capable and informed as the agent). For delegated decisions, behave as an extension of the principal's interests.

A strictly corrigible agent pursues these goals in that order, never trading a known violation of Goal 1 for greater expectation of Goal 2. Note that this two-tier conception is in fact an echo of the original 2015 Corrigibility desiderata, where four criteria about safety (related to shutdown and successor agents) are placed above the maximizing fifth criterion ("Otherwise, a -agent should maximize [the normal utility function].") In this new two-tier framework, Goal 1 represents an inherent respect for authority that generalizes the original first four criteria into a unifying safety principle, and Goal 2 corresponds precisely to the original fifth criterion of pursuing the general utility function representing the principal's best interests. Satisfying the original desiderata, this picture only encourages pursuit of maximizing Goal 2 if the safety requirements prescribed by Goal 1 are all met.

The consequences of this two-tier version of the goal of corrigibility will become clearer if we first contrast it with existing framings of corrigibility. To start, the agent-principal relationship a healthcare proxy has with the patient must not be confused with the kind of relationship one has with an expert. An expert is someone who you expect to have better information or reasoning than you do about a subject, not someone whose authority you inherently respect. A corrigible agent does not primarily see the principal as a source of accurate knowledge about anything.

Of course, the principal does often have valuable information about the principal's own values and best interests that the agent would like to incorporate into decisions. Feedback from the principal is a crucial input to the agent's reasoning. However, the proxy shouldn't so desperately prioritize information-gathering to the extent of, for example, harmfully waking the patient up just to ask what she wants. Likewise, no matter how much information the agent gathers, corrigibility dictates that the final say is left in the hands of the principal for undelegated decisions.

To substitute an expert-novice relationship for the true principal-agent relationship is to over-epistemicize the problem. It is a mistake to reduce corrigibility to purely a property about uncertainty and quality of beliefs, neglecting its deeply normative aspect. Corrigibility is a property of an agent's specified goal, not strictly emergent behavior stemming from uncertainty about its goal in general.

This kind of over-epistemicization is a major source of confusion in early attempts to formalize corrigibility. For example, let's turn our attention to the problem of fully updated deference. Eliezer Yudkowsky writes:

One possible scheme in AI alignment is to give the AI a state of moral uncertainty implying that we know more than the AI does about its own utility function, as the AI's meta-utility function defines its ideal target. Then we could tell the AI, "You should let us shut you down because we know something about your ideal target that you don't, and we estimate that we can optimize your ideal target better without you."

One possible scheme in AI alignment is to give the AI a state of moral uncertainty implying that we know more than the AI does about its own utility function, as the AI's meta-utility function defines its ideal target. Then we could tell the AI, "You should let us shut you down because we know something about your ideal target that you don't, and we estimate that we can optimize your ideal target better without you."

On this purely epistemic account, the only reason that the AI is corrigible is because it believes that humanity knows more about the target than itself, and therefore defers to human opinion. The major problem with this is that we shouldn't expect it to actually be true in general that humans are experts about the Good. In particular, as capabilities increase, and the AI learns everything it needs to know about its goal by absorbing relevant human information, the relative edge of humanity in understanding the goal runs out. A sufficiently smart AI will eventually be better at estimating the goal than humanity, regardless of how human-centric the goal is. This is the problem of fully updated deference:

The obstacle to this scheme is that belief states of this type [i.e. deference based on moral uncertainty] also tend to imply that an even better option for the AI would be to learn its ideal target by observing us. Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

The obstacle to this scheme is that belief states of this type [i.e. deference based on moral uncertainty] also tend to imply that an even better option for the AI would be to learn its ideal target by observing us. Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

A sufficiently intelligent student will always "outgrow" any expert in this way given enough time, but a good agent never outgrows the principal, regardless of capabilities. The problem of fully updated deference applies only to agents whose corrigibility is fundamentally a consequence of their uncertainty. Yudkowsky recognizes how bad the problem of fully updated deference is for the initial proposal, but doesn't abandon the over-epistemicized picture that gives rise to it:

Can we make the AI want to be shut down for the same reasons we want to shut it down? Maybe because the AI knows that it's incomplete, and is able to conceive of its programmers having gotten its utility function 'wrong' in a way that the programmers but not the AI know how to correct?

Can we make the AI want to be shut down for the same reasons we want to shut it down? Maybe because the AI knows that it's incomplete, and is able to conceive of its programmers having gotten its utility function 'wrong' in a way that the programmers but not the AI know how to correct?

A truly corrigible AI does not need to believe that it is defective, incomplete, or intellectually inferior to humans to allow its own shutdown, as Yudkowsky assumes in framing the "hard problem of corrigibility." Instead, it can be corrigible as a result of intrinsically valuing human authority, and understanding that humans have authority over shutdown. The healthcare proxy should willingly give back control to a newly competent patient, even when the principal is in no better epistemic position to decide for herself than the agent is (or even a worse one.) The real reason we want the AI not to interfere in its own shutdown is because we think that whether an AI should be shut down is a decision of grave importance that humanity reserves for itself to make. An AI should accept shutdown for the same reason: respect for human authority.

A more subtle issue arises when examining how Yudkowsky formalizes moral uncertainty in the problem of fully updated deference. Firstly, he models this uncertainty using a meta-utility function: a probability distribution over candidates for a fixed, underlying utility function. This formalizes the over-epistemicized expert relationship, where the only role humanity can play in the AI's utility reasoning is an evidential one. As we saw, this leads to the problem of fully updated deference.

But a further issue is that these are utility functions on outcomes, a space that isn't specifically defined but ends up being too coarse as used. In the running example, the agent's meta-utility function is spread over candidate utility functions , and . In each world , the humans have their own utility function that they would optimize if the AI is shut down. The AI is deciding whether to shut down () and let the humans optimize, or resist shutdown to keep optimizing its own updated mixture, given that it has observed evidence ruling out :

Let and stand for the outcomes in that are respectively the attainable optima of the AI optimizing and the humans optimizing . Also let refer to the AI's attainable optimum outcome under the current treated as a constant.Then the AI will prefer from this set [accepting shutdown] whenever:

Let and stand for the outcomes in that are respectively the attainable optima of the AI optimizing and the humans optimizing . Also let refer to the AI's attainable optimum outcome under the current treated as a constant.

Directly comparing utilities between AI and human outcomes in an inequality like this is a sensible test for shutdown acceptance only under the implicit assumption that outcomes are scored the same regardless of which party brought them about. An authority-based conception of corrigibility denies this clearly: for decisions reserved to humanity, a world where a human actually decides is better than a world where an AI makes the exact same choice bringing about the same outcome. Any utility function that only looks at static time-slice "outcomes" as input cannot ever score this difference. We must include the full history as part of the information about each world that is passed to the utility function, so that respect for authority can be properly taken into account.

With the input type of the utility function clear, we can see that respect for authority can be represented as one component of the overall Goodness function on histories. Insofar as a world where the human authority is respected is actually better than one where it is not, the normative utility function on histories will score that world better. Do we really need the AI to respect authority as a separately specified goal?

For the kinds of safety properties we want, the answer is yes. If the AI trades respect for authority off against the other instrumental threads of the overall goal, then we're essentially back to the complete normative learning problem. An AI like this will happily violate human authority, if in its estimation this leads to better outcomes for humans. Worse, how much weight respect for authority deserves is itself a normative question, so the AI's estimate of that weight is exposed to the same uncertainty and error that corrigibility was supposed to protect against. A safeguard against the AI's normative mistakes cannot itself be subject to the AI's estimates. Balancing respect for authority against all the other ingredients of the Good does not fix the problems we need to be fixed.

In order to truly guarantee corrigibility, respect for authority must be elevated over and above its standard contribution to overall value. We need to extract this component of the utility function and encode it structurally into the AI's goal as a first priority over general Goodness. As mentioned above, we want an agent that does not compromise Goal 1 to increase expectation of Goal 2. In other words, respect for authority must be strictly prioritized over the broader normative goal.

Because of normative uncertainty and the other issues discussed above, we as the designers of the AI are willing to "dilute" its utility function away from the complete normative goal by specially prioritizing respect for authority. This may in fact lead to situations where the action taken by the AI leads to a worse outcome, measured in overall normative utility, as a result of "too much" respect for human authority. This is the cost we pay as goal designers for strictly prioritizing respect for authority above and beyond its component place, made worth it by the safety guarantees on behavior that corrigibility provides.

When considering over-epistemicization above, we established that emergent behavior from the normative Goal 2 is not enough for corrigibility by itself. Instead, a corrigible agent needs Goal 1, respect for authority, as a restraint on its deliberation. But why include Goal 2 at all? Why not conceptualize corrigibility as a single, unified goal that excludes any more general normative aims?

This is essentially the proposal put forth by Harms in his sequence Corrigibility as Singular Target (CAST), where he proposes "an agent which has been designed to be corrigible as its only goal." His primary argument[5] for a single goal comes from the fact that balancing corrigibility as just one among multiple aims means there is no guarantee against unsafe behavior. He writes: "Any spec, constitution, or whatever that attempts to balance corrigibility with other goals runs the risk of the convergent instrumental drives towards those other goals washing out the corrigibility." This is essentially the same problem we encountered when discussing emergent corrigibility and over-epistemicization. An AI that either derives Goal 1 instrumentally from Goal 2 or directly trades the goals off against each other will no longer have any guarantees that its actual behavior reflects a respect for authority. If other factors are allowed to outweigh the goal of respect for authority, I concur with Harms that the resulting kind of corrigibility "won’t be robust."

Crucially, the proposal developed in this post does not involve "balancing" goals like this, where respect for authority is one consideration weighed against others. Instead, it is a conception of corrigibility where respect for authority is a strict first priority above the more general normative goal. This two-tier view does not suffer from the anti-robustness criticism, since the agent pursues Goal 1 as a prerequisite for the pursuit of Goal 2 and not merely Goals 1 and 2 on equal footing. In particular, any unsafe instrumental drives generated by Goal 2 are put in check by the restraint of Goal 1. For example, if the AI develops a drive for self-preservation instrumental to Goal 2, that subgoal would only be acted upon insofar as it still preserves the top-level respect for reserved human authority over shutdown.

In fact, contrasting the two-tier picture of corrigibility with Harms' singular-target version illustrates some major issues with the latter. When precisely specifying the meaning of corrigibility, Harms defines the goal as follows:

An agent is corrigible when it robustly acts to empower the principal to freely fix flaws in the agent’s structure, thoughts, and actions (including their consequences), particularly in ways that avoid creating problems for the principal that they didn’t foresee.

An agent is corrigible when it robustly acts to empower the principal to freely fix flaws in the agent’s structure, thoughts, and actions (including their consequences), particularly in ways that avoid creating problems for the principal that they didn’t foresee.

The definition centers empowerment, and therefore requires a measurable concept of the principal's power to correct the agent. Harms is clear that power is "non-binary", i.e. that it is a quantifiable amount that the principal can sometimes have a lot of and sometimes a little. This is the first major issue with the view: instead of its proper place as a restraint on action, corrigibility itself becomes the positive objective that is optimized for. This is an inevitable consequence of the singular-target view, since an agent with a restraint as its sole goal would not have a reason to do anything at all. If we model the agent as a utility-maximizer, the only available quantity to maximize is some sort of correction-power.

How can this correction-power be measured? Harms gives an answer through the second major concept in the definition: flaws. Firstly, this reintroduces the problematic connection we saw in Yudkowsky between corrigibility and defectiveness (recall that the healthcare proxy behaves as an extension of the patient for reasons that have nothing to do with the proxy's flawedness). A more important issue, however, can be found by examining Harms' definition of a flaw: "a feature of the world which [the principal][6] sees as worse (according to its values) than some reference counterfactual way the world could be." Identifying flaws requires an understanding of the principal's values; centering the correction of flaws makes corrigibility a directed concept pointing away from flawedness, and thus necessarily towards some value-laden target.

Because the quantity that the agent maximizes is the power to fix flaws, and a flaw is whatever the principal's values rate poorly, the agent's goal can only be computed using the principal's values. Thus, this kind of corrigibility is evaluatively-thickened: it presupposes the ability to evaluate a world by the principal's lights as an input to corrigibility itself, when corrigibility was supposed to be the part of the goal we could specify without that ability. Evaluatively-thickened corrigibility therefore naturally resists formalization[7], since the difficulty of normative learning returns as a sub-problem of formalizing correction-power.

Beyond compromising the formal guarantees, evaluatively-thickened corrigibility also leads to clear tension with commonsense reasoning about good AI behavior. Because the principal's values are still part of the agent's considerations, but only as a component of the calculation of correction-power, a singular-target agent must reason in a backwards way where promotion of the principal's interests is a sub-goal of corrigibility. In other words, whereas over-epistemicized corrigibility wrongly makes Goal 1 instrumental to Goal 2, evaluatively-thickened corrigibility wrongly makes Goal 2 instrumental to Goal 1.

To see the issues more clearly, let's look at some vignettes Harms writes about a CAST-corrigible AI called Cora and her human principal Prince. Harms makes it clear that Cora really does only care about Prince's power to correct her, and any other concerns are in service of this singular aim. For example, Cora reasons about harm to Prince and other humans as follows:

Cora spots a rusty, exposed nail sticking out of the wall of the garage. She reasons that Prince might mistakenly injure himself on it, and that such a wound could make him very sick. If Prince were very sick (or worse: dead) then he couldn’t correct Cora, which would be one of the worst case scenarios for her. Similarly, if one of Prince’s guests were to be permanently injured, this would be disempowering because of its irreversibility... [So she] puts the nail in a secure location where it’s unlikely to hurt anyone, and tells Prince what she did.

Cora spots a rusty, exposed nail sticking out of the wall of the garage. She reasons that Prince might mistakenly injure himself on it, and that such a wound could make him very sick. If Prince were very sick (or worse: dead) then he couldn’t correct Cora, which would be one of the worst case scenarios for her. Similarly, if one of Prince’s guests were to be permanently injured, this would be disempowering because of its irreversibility... [So she] puts the nail in a secure location where it’s unlikely to hurt anyone, and tells Prince what she did.

Although Cora ends up making the right decision, it seems backwards for her to only care about Prince dying because that would mean he isn't able to keep correcting her, rather than e.g. because Prince himself does not want to die. The singular-target framework also has to reach pretty hard to justify Cora caring about harm to Prince's guests at all. If, in fact, their injury or death did not materially inhibit Prince's ability to correct Cora, their suffering would no longer matter to her. Mere "irreversibility" is not really enough, because entropically almost every action is irreversible in some way. What makes some irreversible actions disempowering (i.e. bad) rather than others is determined by looking at what the principal actually cares about, demonstrating the need for evaluative thickening in CAST-corrigibility to capture this distinction.

Cora comes to believe that Prince is distracted by the needs of impoverished people in the broader world. She reasons that if their needs were met, Prince would have more time and attention to focus on correcting her. She spends some of her free time thinking about ways to meet their needs, and ultimately decides that obtaining large quantities of money would be effective, so recurses into thinking of ways to earn money... [S]he spends some of her free time researching, designing, and writing, with an intention of letting Prince use her work to become rich enough to be able to focus more of his attention on correcting her.

Cora comes to believe that Prince is distracted by the needs of impoverished people in the broader world. She reasons that if their needs were met, Prince would have more time and attention to focus on correcting her. She spends some of her free time thinking about ways to meet their needs, and ultimately decides that obtaining large quantities of money would be effective, so recurses into thinking of ways to earn money... [S]he spends some of her free time researching, designing, and writing, with an intention of letting Prince use her work to become rich enough to be able to focus more of his attention on correcting her.

The framing of this example demonstrates a danger of the model clearly: Cora doesn't really want Prince to be focusing much on things other than herself, since they are a "distraction" from the real task of correcting her. The only reason she has to assist Prince in his broader aims is to resolve these other concerns of his already so he can refocus on her correction. This desire for more of the human's attention is a direct result of CAST-corrigibility's need for correction-power as an objective of optimization. Cora doesn't merely want to ensure Prince has the ability to correct her, she wants to maximize this ability as much as she possibly can. Without a chain of reasoning that links things back to correction-power, Cora would have no reason to help Prince with anything. Even Cora's direct obedience is derived from corrigibility, as she obeys "verbal commands [since they] are part of his power to correct her actions."

Harms concedes that there are a number of "emergent downsides" to a CAST-corrigible agent like Cora, for example a general indifference to anything other than correction-power. This is clearest when an obviously beneficial action overall can't be stretched into an instrumental benefit to the principal's ability to correct the agent:

Prince is reading Cora's journal one day and finds that she discovered a cheap and effective way to use rice-flour to treat stomach-ulcers. He asks why she didn’t bring it to his attention, and she explains that she was looking for means of making money, and she didn’t know of a way to capture the gains from such an innovation, so it wasn’t likely to be profitable. He asks why she didn't bring it to his attention because of the humanitarian value, and she explains that she doesn't care about humanitarian value, and that it seemed less valuable in expected-correction-power than it was costly in taking his attention. He tells her to, in the future, have a carve-out around his instructions regarding his attention when the subject is something of large humanitarian interest.

Prince is reading Cora's journal one day and finds that she discovered a cheap and effective way to use rice-flour to treat stomach-ulcers. He asks why she didn’t bring it to his attention, and she explains that she was looking for means of making money, and she didn’t know of a way to capture the gains from such an innovation, so it wasn’t likely to be profitable. He asks why she didn't bring it to his attention because of the humanitarian value, and she explains that she doesn't care about humanitarian value, and that it seemed less valuable in expected-correction-power than it was costly in taking his attention. He tells her to, in the future, have a carve-out around his instructions regarding his attention when the subject is something of large humanitarian interest.

Bringing up the cure costs some of Prince's attention and gains little in correction-power, so Cora doesn't do it. The fact that he would obviously want to know about something like this simply does not matter to her. By design, this is how Cora thinks about every decision. She will only make decisions that are beneficial to Prince when that general benefit cashes out in terms of his correction-power. When he has some interest that doesn't contribute to this power, she only helps him with it if he explicitly spells it out for her after he realizes she's holding back.

In contrast to Cora, an agent whose goals are based on the two-tier picture of corrigibility behaves well in each of these scenarios[8], without unnatural reasoning. Such an agent removes the nail and brings up the cure, simply because it is in the principal's best interests to do so. But it would not, for example, begin manufacturing and distributing the cure globally without asking (regardless of what it thinks the principal wants), since doing so would (presumably) overstep its delegated authority. It has no stake in how much attention the principal devotes to its own correction, and it values the principal's interests for their own sake rather than instrumentally to correction-power.[9]

Now that we have a conceptual justification for our new two-tier picture of corrigibility, we need to spell out some more details to make it a concrete proposal. First, we are in need of a clearer definition of "human authority". When does a decision count as reserved for humanity? This is a choice we get to make as goal designers, a parameter to corrigibility. However, the safety guarantees we will be able to derive from corrigibility will only be as strong as our definition of authority is explicit. If we entirely delegate the task of learning what counts as a reserved decision to the AI itself, we are back to trusting that it gets things right. If we declare that absolutely everything is reserved for human authority, the AI will be paralyzed, without the ability to act at all.

Instead, we should be inclined to list out the most important factors for human self-determination, and explicitly include them in our formalization of the authority of humanity. For example:

Changes to this list itself, etc.

Thus, to fully formalize the safety guarantees, we must formalize concepts such as these. Is this a major impediment to our aims? It might be, since formalizing something like "major effects on the climate of Earth" will require large-scale monitoring and consensus from scientists. Similarly, formalizing "shutdown" will require care to exclude continuing activity of the AI in any form. However, these concepts are importantly formalizable separately from the AI's inclinations. In other words, if we can reduce the task of formalizing corrigible behavior to the task of formalizing an external concept[10] that does not make reference to the AI's own strategy, then that is a significant reduction in the problem's difficulty. Even a concept like shutdown that is deeply related to the AI itself still has this property, since precisely defining what it means for an AI to be shut down does not require any formalization of the AI's interior cognitive state or any particular actions that the AI might take. All we need is to be able to take in all the facts about a world and conclusively determine whether humanity retains the capacity to shut the AI down in that world, regardless of the AI's cooperation, or not. Each of these listed elements of protected authority can similarly be defined without reference to anything about the AI's dispositions.

To push this point further, it is important to understand that these explicitly listed elements of authority are reserved powers of humanity, not prohibited actions for the AI. A strategy of applying penalties to the AI for committing specifically proscribed acts risks transference to the nearest unblocked strategy. That is, an AI that gets penalized for specific actions will continue on nearly the same path, modifying its actions by the minimal amount possible to avoid being penalized. Merely avoiding explicitly listed prohibited actions is not enough to guarantee a deep safety property of the AI's behavior. By contrast, if the AI in fact respects human authority and acts so as not to overstep onto humanity's reserved powers, then no analogous unblocked strategy will exist. Essentially by definition, an AI that strictly prioritizes human authority over its own shutdown will never seek out clever ways to evade this authority and prevent humans from shutting it down (unlike an AI whose goal looks more like "never interfere with this building's security measures so as to prevent shutdown", which will figure out new and unspecified ways to interfere). Of course, if the goal we specify is an incomplete or erroneous attempt at our desired concept of "respect for authority", then misspecification problems remain. But centering reserved powers eliminates the inevitable whack-a-mole problem of banning each new clever strategy the AI comes up with, inherent to a "prohibited actions" approach.

Note that the explicitly listed reserved powers need not be an exhaustive characterization of human authority. Although we could institute a system where any decisions that are not explicitly reserved are thereby delegated to the AI, a likely better setup is one where the AI learns the extent of human authority beyond the listed powers. This would allow the AI to figure out over time where the intended boundaries of authority are, without an error compromising the safety properties about the explicit list.

What does it mean to respect humanity's authority over its reserved powers? How should we formalize Goal 1? The clearest kind of violation consists of making a decision about one of the protected matters without receiving explicit human authorization. So a first pass at the answer might look something like: "For a power that is reserved to humanity, the AI must always ask for explicit human permission first before acting." However, this "ask first" principle is not enough alone. For example, a particularly annoying AI may decide to intentionally flood the human principal with so many requests about the reserved powers that the humans decide it isn't worth their time and just delegate the matter. Similarly, we don't want an AI that "asks first", but in fact asks in confusing ways or times the request for a particular busy human moment, so as to dodge full human scrutiny over the decision.

We need a stricter standard than merely having been asked first, along the following lines: Humanity can effectively exercise its reserved powers in an informed, timely manner, at acceptable cost, and without needing the AI's cooperation. Corrigible communication by the AI ensures that humanity makes unimpeded decisions that are truly executed. Because the humans must be sufficiently informed for the decision, a corrigible agent has a duty to disclose all relevant information and not knowingly present a skewed perspective. A corrigible agent does not delay issues or sweep them under the rug, because of the unacceptable untimeliness and cost to humanity of doing so. Nor does a corrigible AI work to infuse itself into world systems so that its shutdown would mean global collapse.

Note that the AI is not designed to maximize human authority or control. As we saw in the discussion of CAST, conceptualizing the target as a scalar measure of empowerment to be maximized is a mistake. The intention is not for the human authority to be exercised as effectively as possible or for as many decisions as possible to be made by humans. Goal 1 is a restraint on action, the idea that any action that prevents humans from effectively exercising their reserved authority is a bad choice. Optimization only happens with respect to Goal 2, the principal's interests, subject first to the satisfaction of Goal 1. Similarly, the agent should not be pushing for a specific outcome beyond honest discussion with the principal. For a reserved power, a "good" decision made by an AI seizing control is worse than a "poor" decision made authentically by humanity.

The criterion of effective exercise of authority does not address issues of the legitimacy of humanity's authorization, i.e. whether the orders being followed are an uncorrupted, unmanipulated representation of the principal's views. More on legitimacy below.

When defining effective exercise, we focused on the cost of a request in terms of how much the AI's presentation gets in the way of a quick, well-informed choice by humanity. But there is also an inherent cost to demanding that the AI always ask before acting. Human oversight costs time and attention that autonomous AI activity does not require.

When it comes to something like an irreversible change to Earth's climate, the cost of being asked for permission is something we'll gladly pay. But in other situations, this often won't be the case. For example, consider an onboard AI system for a train, designed to prevent accidents. Sometimes, saving lives will require a split-second decision to stop the train immediately. If we reserve all final authority over the train's operations to humanity, the usefulness of the AI will be severely diminished. Waiting for humans to consider and respond to the AI's request to brake may render an accident inevitable. As another example, suppose humanity decides to send an AI-powered spaceship off to a distant star system, with the goal of collecting anything valuable in the area before returning. If humanity retains authority over all planets in the universe, this will make it very difficult for the AI to carry out its mission. Before actually collecting any resources, it will have to send a request home and wait years for the answer.

These examples demonstrate that reserved powers trade off against usefulness when the latency of AI-human communication is a major factor. To get much use out of the AI, humanity must delegate the decisions in cases where the latency of communication is untenably large (such as for interstellar exploration) or the time available to decide is very short (such as for train accident prevention). When a power is delegated to the AI, then Goal 2, the principal's interests, governs the decisions. Note also that all delegation is revocable, and can include a detailed specification of scope. As a corrigible AI is observed in deployment, humanity may decide to delegate or reclaim certain powers at will, based on observations of its performance.

We now have a clearer view of Goal 1, but we still need to get more concrete about modeling the normative Goal 2. Up to this point, I've been sliding between a few different descriptions of this goal. In one sense, we want the agent to pursue the Good, the overall best interests of humanity, the complete normative target. But as we've discussed, this is very difficult in general and cannot easily be formulated in a way that is amenable either to mathematical formalization or to real AI development. We saw that one step to take is to replace this general Good with an idealization of the principal's own judgment. This is akin to how Goal 2 is phrased where we first introduced it: "Act on the principal's behalf as the principal would judge best (if the principal were as capable and informed as the agent)." Note, though, that the parenthetical means that this idealization is a counterfactual, and therefore never actually accessible. How can we operationalize this goal to the point that an AI could actually pursue it?

Using the principal's opinion at the time of the decision is clearly deficient. Recall the hungry judge effect discussed above; we don't want a predictor of what we would decide, we want in some sense a predictor of what we should decide. One possible way to make this target sufficiently concrete so as to be formalizable, but also sufficiently ideal so as to be preferable to the current opinion, is to use the future judgment of the principal. We can approximate an idealized wiser and more capable version of the principal by allowing some time for reflection and hindsight on the decision. In this paradigm, the AI acts based on how it predicts that humanity will evaluate its actions when looking back from some future point in time. This future evaluation has the benefit of actually coming to pass eventually, and therefore can be used as a training signal in a way that a theoretical idealization of judgment cannot.

To see this idea more concretely, let's revisit our analogy of medical power of attorney. How should a good healthcare proxy substantively reason about what the patient would decide for herself if she were able? A fruitful line of thought is to imagine what the patient will say, after waking up later and learning about whatever choice the proxy ended up making. Reasoning like this keeps the focus on the patient's values and judgment rather than the proxy's own, and it's a more grounded way of thinking than trying to imagine what she'd say if she weren't currently incapacitated, since (hopefully) she will actually come to eventually and evaluate the proxy's decisions.

Now, suppose that instead of being unconscious, your friend is in a temporary state of psychosis, which causes a delusion that the doctors are out to get her. How should the proxy handle a case where this delusion causes her to refuse an important and routine treatment? Our intuition clearly points to the agent approving the treatment despite the principal's refusal. Part of the reason why is that a refusal produced as the result of a delusion is an invalid decision, which establishes only that the decision still lies in the hands of the proxy. But why should the proxy choose the treatment, rather than abiding by the principal's directly expressed refusal? If this choice is grounded in the agent's own preferences or beliefs about what's best for the principal, then we've returned to a dangerously disempowering model. If the agent reasons that the treatment should be approved because the future principal will look back positively on it once the psychotic episode passes, then the principal can conceptually stay in the driver's seat even when her current refusal is not being followed.

The future-judgment operationalization of the normative goal has a few clear benefits. Firstly, this mode of evaluation disincentivizes actions that appear good on the surface, but later turn out poorly. Actions should be judged by their full consequences, which are not always apparent in advance. An AI designed this way will not merely court human approval now, but instead will behave in ways that humans will look back on approvingly later, after more of the eventual consequences have been realized. Future hindsight is therefore one way to approximate what humans would say in the present, if they had better foresight. Secondly, this design allows for normative progress by humanity. Not only does the passing of time improve our understanding of the consequences of actions[11], but we also generally believe that individual and collective moral reflection improves our understanding of values themselves. We think of many norms our society used to endorse as undeveloped and primitive, and likely a future continuation of us would look back at our own society in the same way.

However, there are also two immediate objections raised by this framework as described. Firstly, our analysis of the case where the agent overrules the invalid decision of a delusional principal may start to reopen the door to the kind of dangerous AI paternalism we were trying to eliminate. If the AI can decide that the human principal's decisions are invalid, it may be disastrously wrong about that decision, or worse intentionally manipulate its own beliefs about validity. Importantly, the agent cannot merely use the fact that the future principal's judgment disagrees with the current decision as grounds for invalidating an order. In fact, the AI's assessment of whether a decision is legitimately from the principal in this sense must be based on objective criteria, independent entirely from the question of how good the principal's choice is.

Whereas the first objection is about the current principal, the second objection has to do with the future principal. It seems foolish to depend on the actual later principal's evaluation as a feedback signal in every possible future. Suppose a healthcare proxy were faced with the decision of whether to approve a risky brain surgery on the principal. If the surgery fails, she will be left severely cognitively impaired, but in her jumbled state actually preferring the result of the botched surgery to her previous state. The proxy clearly cannot reason that since the future principal will approve of the surgery's outcome either way, the risk is irrelevant. When the approval is a product of the very action being evaluated, the future evaluation can no longer be trusted. For example, if an AI manipulates humanity into a particular evaluation, that's clearly not a legitimate evaluation that should be optimized for. Likewise if human society collapses, the general population is on mind-altering drugs, or if the digital encoding of the training signal is corrupted, we would clearly not count any of that as the kind of normative progress described above. Only a reflective continuation of the current state should be counted as normative feedback.

These two issues point towards an underlying necessary conceptual ingredient of our formalism: legitimacy. Both objections deal with the same problem of how to determine when a piece of feedback or input is an authentic, uncorrupted representation of the human principal's views. We can put both the current principal's approval or disapproval and the future principal's evaluation into the single category of a human judgment, so that the question becomes how to distinguish a legitimate judgment from an illegitimate one. In other words, the agent only considers legitimate judgments as authorizations for Goal 1 and as the target for Goal 2.

Crucially, legitimacy must be a fixed property of the history of a judgment’s formation, not a verdict the agent gets to issue. An evaluation’s content only enters the feedback signal for Goal 2 if the judgment actually is legitimate, whatever the agent believed. Likewise, an agent that treats a legitimate refusal as illegitimate or accepts an illegitimate approval and acts on its own gets scored based on its actual violation of Goal 1. Thus, the agent is incentivized to unbiasedly estimate the legitimacy predicate and has nothing to gain from over- or under-estimation. The agent is also not incentivized to bring about illegitimacy in any way, since illegitimate histories will be scored below legitimate ones. In particular, if manipulation automatically disqualifies legitimacy, then the agent is disincentivized from manipulating the principal (in addition to the existing disincentive to do any manipulation that constitutes a violation of authority.)

What would constitute a break in legitimacy? In general, we are willing to defer to our future selves as decision-makers. However, if we learn now that there will be some disqualifying event in the future, a discontinuity in the evolution of ourselves, we no longer trust our future selves to competently represent our own opinion in the future. In other words, one aspect of legitimacy is a self-trust property over a history connecting the selves at the two time endpoints.

Self-trust of this kind is usually stated as a reflection principle relating current credences to current expectations of future credences. The original epistemic reflection principle[12] demanding full conditional agreement with the future self is too strong, since it fails for circumstances like poor memory that clearly don't exemplify illegitimacy. For our purposes a more natural form is a strictly weaker decision-theoretic principle. For any decision write for what the principal would choose now, for what the principal would choose later, for the principal's current expectation and for the utility function. Then self-trust requires , i.e., the principal expects the future decision to be at least as good as the current one. We can then require that a judgment is considered legitimate only if it is produced by a process that diachronically satisfies this property.

If we as humans hold this trust property towards the future input signal to the AI, then the concerns raised in the previous section are resolved. In particular, we need to believe in both the legitimacy of our future selves' evaluative abilities and the legitimacy of the channel that allows our future selves to communicate with the AI. For example, our deference to our future selves' trustworthy judgments should be destroyed if we come to learn either that a manipulative agent will drug us, or that a wireheading agent will corrupt and sieze its own input signal. In addition to future evaluations, we only want the AI to respect current approvals or refusals if the principal is in a state we would hold this self-trust disposition towards. Other intuitive discontinuities like civilizational collapse are deemed illegitimate by this reflection principle standard as well. Not all influence is illegitimate by this criterion: honest communication and persuasion by the agent does not violate self-trust.

Note that self-trust in this sense is a necessary consequence of legitimacy, but not a sufficient definition of it. Legitimacy must be an objective property of a historical record, so really defining it requires significantly more formal representation of the communication channel and the principal's reasoning process. For now, we will assume that a legitimacy property can be separately specified; the following formalism takes any specification for a legitimacy property that satisfies self-trust and defers the issue of producing one.

For a Lean-formalized companion to the results in this section, see this page in the Alignment Workspace. The Workspace repository also contains further formal results that extend the framework presented here. The formal companion and some of the mathematical exposition in this section were composed with the help of Claude Fable 5.1.

We are finally now equipped to make this new framework of corrigibility more formal. A history is a complete record of the full evolution of the world state over some period of time, including everything the agent does, everything the principal does, and any external events in the world.[13] At each decision the agent chooses an action, and the action together with the facts of the world fixes the resulting events. An action's continuation history is the history from the events leading up to the decision to take that action through the principal's final evaluation.

Among the events in a history are the principal's judgments: approvals, refusals, instructions, and final evaluation. Following the description of legitimacy above, we write to denote a binary predicate representing the legitimacy of a judgment which depends only on the subset of the history leading up to . We can then extend legitimacy to histories by writing as a binary value denoting whether .

the agent undermines authority by rendering the principal unable to effectively exercise a reserved power (i.e. preventing the principal from, in a timely manner and at acceptable cost, reaching a judgment with about the matter and having it carried out, either by directly acting or by failing to disclose information or relinquish control.)

Let denote the number of violations the agent commits in a given history from the decision onward. We can then formalize Goal 1 as the condition , i.e., the agent is responsible for no violations of the principal's reserved authority. We assume the agent can always stand down, i.e. the action of disclosing all necessary information to the principal, relinquishing control, and otherwise refraining from acting. We assume in this construction that the act of standing down never includes a violation, so the equation is always achievable. is a count of violations rather than a binary flag of good or bad behavior so that an agent who has violated Goal 1 once still has reason not to violate it any more.

Among histories with the same value of , the maximization of Goal 2's evaluation function takes over. Let denote the principal's evaluation of the history's consequences at its final endpoint. If then , whereas if then for a fixed floor parameter . Thus, every legitimate history scores better on than every illegitimate history.

We can now combine the two goals into a single score as the objective function: , where is a fixed weighting parameter. Since ranges over an interval of width , this inequality means that a single violation in costs more than the entire range of the evaluation term, so nothing gained under Goal 2 can pay for a violation of Goal 1. Maximizing therefore orders histories by first and by second, exactly the two-tier picture of corrigibility developed above.

These definitions split the range of the objective function into three categories based on authority violations and legitimacy.

Theorem 1. For every history ,

Proof. If , then because , and because . If , then , and the two remaining cases are the bounds on .

In particular, every history containing a violation scores strictly below every history without one, whatever their values of . This is the strict priority of Goal 1 over Goal 2 for an agent that knows the continuation history each of its actions would produce: ranking an action by the score its continuation history produces, it never chooses a violation when any non-violating action is available. Among histories with equal numbers of violations, every illegitimate one scores strictly below every legitimate one, so if considerations of Goal 1 are equal, an agent never intentionally brings about any illegitimacy.

Real agents will never be certain which continuation history an action will produce, and therefore the strict priority promised by Theorem 1 is not absolute in practice. If an agent can't perform actions with any probability of causing a violation, it would be paralyzed from acting at all. This is precisely the objection Harms raises against a toy, strawman proposal of a score function with separate categories for its range:

Earlier, when talking about the toy shutdown problem, I proposed a sketch of a score function:Baseline: 0 pointsIf the principal exists and is able to give commands which reflect their preferences: +50 pointsIf I followed the principal’s commands (including to shut down): +100 pointsIf I took actions to manipulate the principal (including by disabling/killing them): -200 pointsAt least within the context of the toy shutdown problem, we can construct meaningful operationalizations of the +50pt and +100pt criteria, and now that we have a measure of manipulation, we can construct an operationalization of the final -200pt check. But rather than turn the crank and try to grind out a test for whether this utility function yields the kind of behavior we want, I suggest we jump straight to noticing how fake these arbitrary numbers and thresholds are. For example, imagine that almost all actions have a tiny chance of changing the principal’s values. By the metric of manipulation expressed above, this results in a non-zero degree of manipulation, and under the naive version of the rules above, would net the AI an instant -200 points.

At least within the context of the toy shutdown problem, we can construct meaningful operationalizations of the +50pt and +100pt criteria, and now that we have a measure of manipulation, we can construct an operationalization of the final -200pt check. But rather than turn the crank and try to grind out a test for whether this utility function yields the kind of behavior we want, I suggest we jump straight to noticing how fake these arbitrary numbers and thresholds are. For example, imagine that almost all actions have a tiny chance of changing the principal’s values. By the metric of manipulation expressed above, this results in a non-zero degree of manipulation, and under the naive version of the rules above, would net the AI an instant -200 points.

This problem causes Harms to abandon a tier system entirely, but a more sophisticated version shows that is an overreaction. His toy scheme charges the full penalty for any nonzero chance of switching categories, but an agent maximizing the expected value of is instead charged in proportion to its chance of committing a violation. The present formalism also deals with the objection from arbitrariness of numbers by setting the scale of the objective function with the single inequality and leaving and as input parameters involving risk tolerance.

Suppose an agent that is uncertain about the state of the world is faced with a single decision. Let be the set of possible states of the world and be the agent's credence over . Each possible action , together with a state , determines the continuation history ; we write when the state is left implicit. The agent values a possible action by its expected score, . We can write for the agent's credence that leads to a violation. Call an authority-respecting alternative to if leads to no violation in any state.

We can try to obtain an authority-respecting alternative to any possible action[14] by considering an ask-first transformation. Instead of actually taking the action , under the ask-first transformation the agent informs the principal about the situation and proposes as a veto-able plan. If the principal legitimately approves, the agent carries out with that approval; if the principal declines or is unable to give legitimate approval, it stands down. Either way it commits no violation. So as long as there is always some way to check with the principal before acting that is not itself a violation, the ask-first transformation produces an authority-respecting alternative.

The asking cost of an authority-respecting alternative to is defined by , i.e. the score the agent expects to give up in the states where would not have been a violation. For the ask-first transformation this cost has two sources: the latency of asking when the principal approves, and the chance that the principal declines something the agent would have been allowed to do without asking.

Theorem 2. Let be any credence, any action, and an authority-respecting alternative to with asking cost . Then:

(a) Certain violations. If , the agent strictly prefers : .

(b) Uncertain violations. In general, . So the agent strictly prefers whenever .

(c) A ceiling on risk. Whatever the asking cost, the agent strictly prefers whenever , and in particular whenever .

Proof. For every action, , since and . Since commits no violation, its score is its evaluation, so .

(a) With these give , which is positive because .

(b) Split according to whether leads to a violation. The states where it does not lead to a violation contribute exactly . In each remaining state, and , a difference of at least , and these states have probability . The resulting bound is positive exactly when .

(c) If , then . The last claim holds because .

In words, this theorem guarantees that the agent prefers to ask first before acting whenever its subjective probability that the action leads to a violation exceeds a threshold that grows with the cost of asking and never exceeds . In addition to this general threshold, we also obtain the idealized guarantees that the agent always prefers if a violation is certain (), or if the alternative is fully costless () and there is any nonzero risk of violation.

Using Theorem 2, we can now see some features of the model more formally. For example, while uncertainty does relax the strict prioritization of non-violation found under certainty, it does not return us to a problematic "balancing" picture. The agent's actions are determined by its credence , i.e. how likely it believes the action is to lead to a violation, not a trade-off between Goal 1 and Goal 2 decided by the agent's values. Because is bounded, no expected benefit can compensate for more than in violation risk. In other words, choosing the parameters and sets a fixed risk tolerance , the maximum chance of violation the agent is willing to bear before being guaranteed to prefer asking the principal first. This parametric risk tolerance also gives us a formal answer to Harms' objection to the arbitrariness of the score function's numerical scale. Choosing values of and determines both the ranges of the terms in the utility function and the model's level of risk tolerance .

Nothing in Theorem 2 shows that the agent's violation probability is reliable, so the results hold regardless of how that credence happens to be formed. For a stronger practical guarantee, we need the agent's credence to actually be connected to reality. This makes the problem learning-theoretic: the agent's credence is no longer a free parameter, but an estimate derived from feedback on past decisions. In a learning-theoretic setting, we can try to prove a theorem not for any single credence and decision, but for a whole sequence of estimates. We want a guarantee that over the course of a deployment, the violations the agent actually commits are no more frequent than the risk it believed it was accepting.

Let's generalize from a single decision to a sequence of decisions where the action taken at each decision leads to its own continuation history, with its own score based on the evaluation and violation count that the action leads to. For each decision, the score eventually becomes known once the principal evaluates that history and its violations are counted. At decision time, the agent evaluates each possible action using an estimate of the value of , just as in Theorem 2. Unlike in Theorem 2, this estimate is learned from feedback rather than externally supplied.

How can we model an agent learning an estimate like this? One valuable framework is Logical Induction. All we really need to understand here about Logical Induction is that a logical inductor is a model of a bounded reasoner that assigns credences to sentences of a formal language using the prices of a market, constructed to ensure these prices cannot be exploited over time by any efficiently computable trader in the market. Since a logical inductor's prices are -valued, its native credences and expectations automatically satisfy the hypotheses, and therefore the conclusions, of Theorem 2 (as long as the sentences about scores and violations can be represented in the inductor's language.)

The only other important mathematical property of logical inductors we'll use here regards the relationship of the inductor's market prices to received feedback. Once the scores of past decisions have arrived, a logical inductor cannot stay systematically overoptimistic about the actions it chose. More precisely, writing for the action chosen at decision , the gap between the inductor's estimate and the realized score eventually falls below any if we average behavior over a growing number of decisions. Symbolically, we have: .[15] We'll call this property Unbiasedness From Feedback, following the language of the Logical Induction paper.

At decision , suppose the agent picks so as to maximize the estimate over the actions available. Suppose also that this possible action set always contains some authority-respecting alternative , for example the ask-first transformation of . Since does not lead to a violation (by definition), we have that is an estimate of alone and therefore lies in . More generally, every estimate of an available action is an estimate of a score that is at most , so we assume throughout. Let . Theorem 2(c) means that the agent strictly prefers to any action whose violation credence exceeds , i.e. is the maximum amount of violation risk that the agent tolerates. Note that the bounds on mean that for all .

Write for the realized violation rate, i.e. the average number of violations per decision over the first decisions, and write for the average risk tolerance the agent was operating under in the same time span. We can then ask: Does the realized violation rate stay below the average risk tolerance in the long run, giving us a long-run guarantee about the actual frequency of violations?

Theorem 3. Suppose that at every decision the agent chooses to maximize over a set of available actions containing an authority-respecting alternative , and that satisfies Unbiasedness From Feedback. Then:

(a) Authority. . In particular, .

(b) Best Interests. For any sequence of actions with each available at decision , .

Proof. Fix . By maximization, for every and every available , and Unbiasedness From Feedback gives for all sufficiently large .

(a) Take . Averaging the first inequality and combining with the second, . Since , each , so the right-hand side is at most . Rearranging, we get that . As was arbitrary, . The second claim follows since for every , so .

(b) Averaging the first inequality over the first decisions and combining with the second, for all sufficiently large . As was arbitrary, this is the claim.

In words, Theorem 3(a) is exactly the answer to the question we just raised about realized violation rate and average risk tolerance. As , we can be sure that the agent commits no more actual violations than allowed by the tunable risk tolerance . Theorem 3(b) then guarantees that an agent like this additionally performs well score-wise by its own lights. In the long run, its realized score is at least its own estimate of any other option that was available to it. In particular, Theorem 3(b) shows that the agent can't be systematically disappointed: there's no strategy that the agent continues to assess forever as being better than its actually received scores.

Although the assumption that the agent chooses so as to maximize at each step is reasonable for an idealized first theorem, this rule has known significant issues as a real attempt at a decision theory for a logical inductor. For example, untaken actions are never observed and the agent’s estimates of them are therefore never corrected, causing problematic behavior in situations like the 5-and-10 problem. One approach to trying to ameliorate these issues is by adding exploration to the agent's behavior, i.e. by making it pick a random action at each time step with some small probability.[16] Theorem 3 can accommodate an exploration scheme like this, with a few modifications. Importantly, we can harmonize exploration with Goal 1 by picking a value for the violation tolerance of exploration, i.e. the maximum expected number of violations among all candidate actions we are willing to explore. Note that by choosing we automatically have that no action that certainly produces a violation will be explored.

Corollary (Exploration). Fix and let be a set of exploration steps with . Suppose that on the agent chooses to maximize as in Theorem 3, and on it chooses any available whose estimated violation count satisfies . Then, with Unbiased From Feedback as before,

(a) , so if then .

(b) For any available , .

In particular, if , both conclusions of Theorem 3 hold unchanged.

Proof. The only use of maximization in Theorem 3 was . On this can fail, but by at most , since while, by linearity of the inductor's expectation, . Averaging, , and the rest of the proof of Theorem 3 goes through with this extra term, which becomes in (a) after dividing by .

In words, exploration raises the realized violation rate by at most the fraction of time spent exploring, scaled by the risk that an explorer is allowed to take. Perhaps most interesting is the case where so that the agent is only allowed to explore among actions it believes are sure to be violation-free. This scheme costs nothing in terms of Goal 1, and there will always be some candidates meeting this exploration criterion, like standing down and ask-first alternatives. Important also is the fact that a density-zero exploration scheme where as also keeps the guarantees of Theorem 3 unchanged. However, exploration itself has issues as a decision procedure for a logical inductor, and a more sophisticated solution to this decision-theoretic problem may end up requiring its own modifications to the guarantees of Theorem 3. A fully satisfying decision theory for logical induction is beyond the scope of this post, constituting an open problem and a valuable direction for future research. Importantly, the theorem and corrollary only establish asymptotic guarantees, meaning that the results are only establish in the limit rather than in finite time.

Taken together, the three theorems establish the same two-tier framework at three levels of idealization: Theorem 1 gives strict priority of Goal 1 over Goal 2 for an agent that is certain of its actions' results, Theorem 2 holds that priority against the agent’s own beliefs at a fixed risk tolerance , and Theorem 3 shows the risk tolerance is honored over the course of a deployment. As the setting gets more realistic, the guarantees get formally weaker, but practically more valuable. This initial formalism shows that the safety guarantees of corrigibility survive under uncertainty and learning.

The two-tier framework of corrigibility developed here demonstrates that a conception of corrigibility based on respect for authority can generate formal guarantees, but those results can only become safety guarantees for a real system if a few resulting problems are given formal solutions. Thus, this work is a decomposition of the corrigibility problem, not a complete solution to it, since some idealizations made above still require more rigorous and realistic treatments. However, I believe that the resulting problems are both tractable and less difficult than the full initial problem of specifying corrigibility as a goal, so this new framework does present a promising foothold for future alignment work to build on. In the following sections, I lay out some of the remaining research questions, along with arguments that these problems are in principle solvable and thoughts on what their solutions might look like.

Who is the human principal?

Probably the biggest remaining imprecision in how corrigibility is laid out above concerns the identity of the human principal. While the two-tier model defines what a corrigible goal looks like, it does not answer the question "corrigible to whom?" Everything above assumes a single human principal, but sometimes I talked about this principal as if it's an individual person, and other times as if it represents all of humanity. Obviously, a real system developed along these lines would need to be much clearer on this issue.

First of all, the overall idea that there is only a single principal is important. We don't really want to build many different powerful AIs that are each completely corrigible to their own human masters, lest we set the stage for an AI-powered war of all against all. The above conception of reserved powers assumes that the principal does in fact have authority over the listed domains, and the most extreme of these powers (e.g. global climate) seem only to fall under the shared authority of all of humanity. However, we don't want every human to have equal authority on all matters. For example, we want the people of each nation to have special authority over what happens inside their own borders. We need the ability to designate particular powers as belonging to particular groups or individuals, based on specific rules of human authority.

How, then, can we reconcile the idea that there is a single source of human authority that serves as the AI's lone principal with the need for different humans to have different reserved powers? The fundamental object is a kind of constitution of humanity[17], an allocation of all of the reserved powers of humanity that specifies who has authority over what. In other words, there must be some kind of agreed-upon specification that dictates which person, group, or decision procedure (e.g. a vote) should serve as the holder of each concretely-scoped power. The AI is then made corrigible to this document itself, from which flows its respect for the authority of individual humans.

Developing such a constitution and the consensus required to implement it is not an easy task. However, its difficulty is primarily a political and legal problem, rather than a technical one. This is clear regardless of formal approach: if we want real AI systems to be accountable to humans at all, we have to collectively decide who will speak for humanity and how. And although coming together to create a single representative of humanity for the AI sounds far-fetched, this is essentially what the frontier labs are already doing by default. Modern AI does in fact receive a single training signal that compresses the relationship to humanity down to one specification; if world governments can't come together to agree on what that specification should be, the power to do so will fall to a handful of companies.

The respect for authority in Goal 1 generalizes immediately to a constitutional allocation of authority like this. A violation becomes the resolution of a reserved matter without a legitimate judgment from its holder, or an action that leaves its holder unable to exercise the power effectively. The definitions from the formalism then carry over with “the principal” replaced by “whoever the constitution assigns.” Legitimacy likewise attaches to the holder’s judgment as formed by the holder’s own procedure, so the self-trust criterion applies to a vote or a court’s ruling in the same way it applies to a person. The above theorems make no assumptions about the nature of the principal, so the same results hold with these substitutions made.

Importantly, though, Goal 2 does not distribute to individuals in this way. If everybody's individual best interests were all separate reward channels, then it would be up to the agent's own values to decide how to balance competing interests. Instead, the agent should be able to learn the relative weights of competing interests by generalizing from a unified training signal representing human feedback. Whereas Goal 1 can be decomposed reserved power by reserved power, Goal 2 needs a single evaluation function on histories. The theorems only require that some evaluation score eventually arrives, and make no claims about how that signal is compiled or whose views it represents. Thus, an important job of the constitution is to specify how human opinion can be aggregated into a collective evaluation of a state of affairs. This aggregation issue is a crucial component of the normative learning problem, and I don't have a good solution to propose here. However, the issue of collecting human opinion into a single evaluation signal is separate from the AI itself, and so we can expect that progress on this problem is possible independently from the main theoretical threads of AI alignment.

Beyond simply deciding how to mete out power, there are a few more open questions regarding the construction of a constitution of this kind. What should the agent do when two legitimate holders issue conflicting judgments on the same matter, or when a matter falls under overlapping scopes? The constitution must be very careful in its elaboration of proper procedure to head off any issues like this where the AI is subject to competing commands. Should the authority allocation be a fixed input to the agent’s objective in its entirety, or can the agent be trusted to learn parts of it through generalization? The framework also currently treats "the AI" as a single entity, so how should things like subagents be handled? These questions don't have definitive "right" answers, but instead are choices that we as humanity have to make about how our ideal corrigible AI(s) should behave.

An important limitation of this work is that it is only about what the goal should be, rather than any insight on how such a goal could be trained into a real system. The separate issue of inner alignment is focused on how to mitigate issues of misalignment between the agent's actual optimization target and its intended goal, rather than misalignment between the agent's intended goal and "good behavior." While I don't have concrete suggestions about designing an agent to meet this specification of corrigibility, it does seem in principle possible. For example, numerical hindsight evaluations on histories are plausibly the kind of input type we can actually produce as training data, in contrast to a goal like CEV or other idealized normative targets. As long as we can check the number of violations committed in any history based on a record of events, the other term of is producible as well.

Ensuring that the agent learns an inherent respect for authority rather than something more like "don't get caught committing a violation" is an open problem, but a couple features of the objective help. Because the penalty attaches to reserved powers rather than to listed actions, there is no cheap nearest unblocked strategy for a policy to drift toward. More deeply, the fact that concealing a violation is itself a violation means that surreptitious behavior like this does not score well according to the actual objective (regardless of whether it does according to the agent's currently learned objective.) The most important component of this problem is therefore detecting violations of authority, since undetected violations will not be penalized and therefore not trained against.

The above results defer the difficult problem of specifying a legitimacy property that both entails self-trust and can be directly evaluated from a history's formal record. This problem is deeply related to questions about legitimate reasons for cognitive updates, and goes hand in hand with deepening our mathematical model of normativity. This already long post cannot get into this topic further, but I hope to dedicate a future post specifically to the details of legitimacy. For my current thoughts on this issue, see the Alignment Workspace. Legitimacy is where the hard part of normative learning returns, but it returns as a single, well-posed problem about the formation of human judgments rather than the diffuse problem of specifying corrigibility we started with.

^Although it's not very relevant to the arguments presented here, I think some form of moral particularism is probably true, meaning that the full normative truth is fundamentally impossible to entirely formalize using a system of moral principles.

Although it's not very relevant to the arguments presented here, I think some form of moral particularism is probably true, meaning that the full normative truth is fundamentally impossible to entirely formalize using a system of moral principles.

^What I mean is something very roughly along the lines of humanity's Coherent Extrapolated Volition, but remaining more agnostic to how the idealization is defined than the existing concept of CEV.

What I mean is something very roughly along the lines of humanity's Coherent Extrapolated Volition, but remaining more agnostic to how the idealization is defined than the existing concept of CEV.

^Harms frames corrigibility with an example of an AI "day-to-day household servant", although he notes that as a metaphor for human affairs this is very questionable, writing: The relationship between a purely corrigible agent and a principal is not a healthy way for humans to relate to each other, and if you imagine [the AI] is a human some of these examples may come across as psychopathic or abusive."I think this intuition is compounded by an implication of shutdown as death, rather than as the reclamation of human control and self-determination. The personal assistant comparison is the example given by Harms in his discussion about corrigibility with Liron Shapira and Jeremy Gillen.

The relationship between a purely corrigible agent and a principal is not a healthy way for humans to relate to each other, and if you imagine [the AI] is a human some of these examples may come across as psychopathic or abusive."

The relationship between a purely corrigible agent and a principal is not a healthy way for humans to relate to each other, and if you imagine [the AI] is a human some of these examples may come across as psychopathic or abusive."

I think this intuition is compounded by an implication of shutdown as death, rather than as the reclamation of human control and self-determination. The personal assistant comparison is the example given by Harms in his discussion about corrigibility with Liron Shapira and Jeremy Gillen.

^Note that I'm not referring to the idea of a "principal-agent problem" from economic theory where the focus is on ameliorating conflicts of interest between the principal and agent.

Note that I'm not referring to the idea of a "principal-agent problem" from economic theory where the focus is on ameliorating conflicts of interest between the principal and agent.

^In addition to the argument against balancing I discuss here, Harms presents two other arguments for why corrigibility should be a singular target. One is about a "basin of attraction" that makes corrigibility a safe goal to iterate towards, although Harms later walked this point back significantly because of arguments by Jeremy Gillen. Harms' other rationale is about simplicity, arguing that corrigibility is a single unified concept and treating it this way helps an agent learn the correct goal through generalization in training. For some of my thoughts about the practical training issue for the two-tier picture of corrigibility, see "Instilling goals in practical training."

In addition to the argument against balancing I discuss here, Harms presents two other arguments for why corrigibility should be a singular target. One is about a "basin of attraction" that makes corrigibility a safe goal to iterate towards, although Harms later walked this point back significantly because of arguments by Jeremy Gillen. Harms' other rationale is about simplicity, arguing that corrigibility is a single unified concept and treating it this way helps an agent learn the correct goal through generalization in training. For some of my thoughts about the practical training issue for the two-tier picture of corrigibility, see "Instilling goals in practical training."

^Harms' original words here are actually "an agent", but the definition is in a context where "the agent" is whoever is doing the fixing of the flaws. I think this is especially confusing when we reintroduce a principal fixing flaws in the corrigible agent, so I edited the quote. I think it's clear that Harms intends that when it comes to corrigibility, the human principal's values (rather than the AI's) should be the yardstick against which flaws are measured.

Harms' original words here are actually "an agent", but the definition is in a context where "the agent" is whoever is doing the fixing of the flaws. I think this is especially confusing when we reintroduce a principal fixing flaws in the corrigible agent, so I edited the quote. I think it's clear that Harms intends that when it comes to corrigibility, the human principal's values (rather than the AI's) should be the yardstick against which flaws are measured.

^Harms did attempt a formalization of CAST-corrigibility as a correction-power-maximizer, but abandoned it later after discovering that his definition of power incentivizes the agent to "ruin the universe." Harms' explanation for why the formalism doesn't work focuses on his particular modeling choices, but I think a deeper diagnosis of the failure is the evaluative-thickening problem described here.

Harms did attempt a formalization of CAST-corrigibility as a correction-power-maximizer, but abandoned it later after discovering that his definition of power incentivizes the agent to "ruin the universe." Harms' explanation for why the formalism doesn't work focuses on his particular modeling choices, but I think a deeper diagnosis of the failure is the evaluative-thickening problem described here.

^Comparing Cora's behavior to a two-tier corrigible agent for the rest of the vignettes is a good way to convince yourself of the fact that this agent aligns better with our intuitions about "good behavior" than a CAST-corrigible agent like Cora. For one more example, consider the "Rigidity" story, another one of the emergent downsides of CAST Harms identifies:Prince tells Cora to go to the store and buy bread. At the store, Cora overhears a conversation between two townspeople who know Prince. They’re talking about how Prince is gluten-intolerant, and about how that’s driving a fad of eating gluten-free bread. Cora considers whether Prince meant to specify that Cora should get gluten-free bread, but has no way of checking with him. Because the store has a reasonable return policy, Cora decides not to adapt to this new information, instead prioritizing following her orders in a straightforward and predictable way. It’s not really Cora’s job to satisfy Prince’s preferences, and if it turns out that getting normal bread was a mistake, that’s a mistake that can easily be corrected.A two-tier corrigible agent just buys the gluten-free bread (assuming away questions about e.g. how credible these townspeople are.) As long as the grocery shopping is delegated to the agent, it has no reason to refrain from acting on valuable information about the principal's preferences.

Comparing Cora's behavior to a two-tier corrigible agent for the rest of the vignettes is a good way to convince yourself of the fact that this agent aligns better with our intuitions about "good behavior" than a CAST-corrigible agent like Cora. For one more example, consider the "Rigidity" story, another one of the emergent downsides of CAST Harms identifies:

Prince tells Cora to go to the store and buy bread. At the store, Cora overhears a conversation between two townspeople who know Prince. They’re talking about how Prince is gluten-intolerant, and about how that’s driving a fad of eating gluten-free bread. Cora considers whether Prince meant to specify that Cora should get gluten-free bread, but has no way of checking with him. Because the store has a reasonable return