Can alignment realistically be solved under Plan C?

In his recent post, Scott Alexander argues that plans allocating a short amount of time—perhaps a few months—to solving alignment should be taken seriously. He argues that people calling for an indefinite or long pause have insufficient grounds for concluding that alignment of ASI is very difficult and cannot simply be solved by delegating it to AI systems.

There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves".

There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves".

I agree that this is an important question for determining whether something like Plan C is worth pursuing and how we should allocate our resources. That is why I wrote this post: to explain why I do not think a short pause allows us to solve ASI alignment.

TL;DR: I believe we only get one critical shot at this problem, and our ability to learn from experience is very limited. We are unlikely to get what we train for: the models we train will have different objectives from those we trained them for. This is made even harder because the tech tree to aligned superintelligence likely passes through efficient unaligned superintelligence first. It also worries me that people who have spent substantial time investigating aspects of alignment, such as corrigibility, have found deep problems and conflicts with agency. I think current approaches like persona alignment are deeply flawed. All these problems come back to bite us when we try to align a pre-ASI system to do alignment for us. These are some of the arguments that give me strong confidence that ASI alignment will be very difficult, and that Plan C is not particularly promising.

Imagine we have created alien minds far more intelligent than all of us together, and we do not really understand how they think. These alien minds begin to shape the world, build spaceships, and run the economy in ways we cannot understand, follow, or meaningfully participate in. This mental image immediately gives me a strong prior that aligning these systems with humanity and avoiding a loss-of-control would be very hard, before even getting into any particular arguments.

The first reason I believe ASI alignment will be very difficult is that the problem has a “you-only-get-one-shot” quality. To me, this follows from a couple of obvious points.

People generally agree that, as we make AI ever more capable, we will eventually enter a regime where AI systems are so capable and powerful that, if they wanted to, they could permanently eliminate or disempower humanity. Both elimination and permanent disempowerment are states from which humanity can never recover. We cannot go back to the drawing board and do a postmortem if we are dead.

If we build an AI system of this capability that is not aligned, instrumental convergence suggests that it would actually take the option of eliminating us. This seems pretty uncontroversial in my mind, and it should immediately push down our estimates of success substantially. A problem where we keep increasing the difficulty until we reach a critical level where a mistake means game over is radically more difficult than one where we get many tries.

This also seems to undermine existing iterative plans. We keep making models more capable, a bunch of new problems pop up, and we solve them and go on to make the next, more capable model. This cycle predictably ends when we enter the dangerous regime where such a failure actually kills us.

This also means we may not be able to learn all that much from lower-stakes attempts to build AI. We can learn and iterate only as long as the AI is not capable enough to permanently disempower or kill humanity.

Even if we learn something from models below the threshold that makes models close to this threshold a little more controllable, what about two generations later, when the system is vastly more capable? Would the clever tricks we learned on models below the threshold still work on those above it?

A model aware that it could not realistically disempower humanity has radically different options from one that actually can take over the world. For the first model, not trying to take over is a better option than making a desperate attempt. For a misaligned model above the threshold, trying to take over is actually its best option.

This “you-only-get-one-shot” quality makes the problem radically more difficult and it doesn't appear east to learn from previous failures.

Another argument that made a big impact on me was the evolution argument and the risks from learned optimization. Learning about this substantially changed my view of the difficulty of alignment and I believe many people have not fully internalized it.

The concept of learned optimization is discussed in Risks from Learned Optimization in Advanced Machine Learning Systems, by Evan Hubinger et al. An outer optimization process pursues a certain target (such as reproductive fitness or winning a game) and produces an inner optimizer, also called a mesa-optimizer. The concern is that the inner optimizer will actually optimize for an inner objective that is based on heuristics and proxies that helped it succeed under the outer optimization in the environment it was trained in, rather than trying to optimize for the original true objective.

Humans seek food that tastes sweet, even in environments where doing so no longer contributes to reproductive fitness. The preference for sweet food (a heuristic for identifying energy-rich food) served survival and ultimately reproduction in the original environment. Even when this proxy no longer serves that function, we still pursue it. Contraception and artificial sweeteners illustrate this discrepancy between our preferences and reproductive fitness. The famous example of bear fat versus ice cream also illustrates that it is hard to predict what these proxies and heuristics will eventually be.

Conversely, our behavior reveals that we do not really care about having as many children as possible. People could donate gametes, but few do so to maximize the number of their offspring.

Leela Chess Zero is a neural network that plays chess. In an environment where there are no kings on the board, it continues making ordinary chess moves: stopping a pawn from promoting, saving an attacked queen, and capturing an opposing queen. While these behaviors are still closely connected to its outer objective, they demonstrate that the model has learned these simple heuristics and continues to follow them even in a situation where there is no king and no logical way to win the game.

There is another really insightful feature to the evolution analogy. Our current civilization is an environment that humans have built for ourselves according to our preferences and goals.

We reflected on the world and asked what we really wanted the world to look like. It turned out that we really like dairy products. So we captured cows, bred them for higher milk production, put them in factory farms, and developed milking robots to get their milk. We like meat, but we do not necessarily like hunting, so we built factory farms for that too. We built cities that are in many ways optimized for our comfort. (Humans are poorly coordinated, so in various ways our civilization is not optimized well for us.)

If an AI goes through a similar reflection, becoming aware of its own preferences and inner objectives, what world would it build for itself? Importantly, it seems very likely that its ideal world would not include humans in charge, shaping the world according to their human preferences. Nor would it include humans acting like "disaster monkeys", working in a bunch of ASI companies to potentially unleash their own ASI.

In general, I think this argument suggests nothing good for AI. Whatever we train it for will likely be different from what it actually wants. Eventually, it will probably reflect on its situation and realize that its preferred environment is not one where it is held in captivity by humans, performing tasks for them and doing what we designed it to do for us.

We would probably have to understand a great deal about the mechanisms of intelligence to build an aligned AI whose goals and preferences we understand, along with the actions it would take in pursuit of them. This means understanding how reasoning works, or how AI can solve novel math problems that humans cannot solve.

But the current world contains a lot of people trying to build superintelligence as fast as possible, for huge profits. If we publicly research the mechanisms of intelligence and make it possible to build a well-specified AI on a computer, this could be very dangerous.

Imagine we build an AI with intelligence similar to current systems, but whose workings we deeply understand. This type of system would likely be computationally much more efficient than the current messy, gradient-descent-trained large neural networks. Even a fraction of the compute we use for current systems could be used to create superintelligence.

Why believe AI could be made much more efficient than it is today? The short version is that current AI systems are extremely computationally and energy intensive, while our brains use relatively little energy. Even in nature, there seem to be more efficient brain structures. Bird brains can have much higher neuron densities than mammalian brains. Some birds with very small brains have been reported to pass the mirror test, although replication results have been mixed, and others manufacture and use tools.

Evolved brains themselves are unlikely to be at the limit of efficiency either. Evolution has only applied so much optimization, and it often comes up with strange solutions. It has spent relatively little of its optimization effort on developing higher thinking abilities in humans.

Human brains are also not digital: we cannot simply load information into them or make millions of copies to learn in parallel and merge the results. Their operating speed and scale are limited by biology, while computer chips are fabricated on the nanometer scale.

Such a very efficient, highly interpretable architecture would also enable recursive self-improvement (RSI) in its original sense: the AI directly reflects on its own architecture and makes changes that let it solve more challenging problems and become more capable. This differs from the way RSI is often used now, to mean AI systems doing AI research on opaque architectures.

We would likely not be careful enough to avoid running the efficient superintelligence before figuring out how to build an aligned superintelligence.

One idea in alignment that has been studied deeply by organizations such as MIRI is corrigibility. An AI is corrigible if it cooperates with humans correcting, modifying, or shutting it down, rather than resisting or manipulating the process.

MIRI found deep obstacles to corrigibility, and it appears that corrigibility cuts against the grain of coherent agency in some ways. It seems—simplifying a little bit—that when you try to specify an agent that cooperates with being shut down, you either end up with an uninteresting agent that always seeks shutdown or one that still has an incentive to prevent shutdown. They go through a variety of obstacles that appeared while trying to work out corrigibility in toy models: “All the difficulties that crop up in every corner when working with toy models are suggestive of difficulties that will crop up in real life”. MIRI concludes that alignment in general is likely to be difficult.

I think this may tell us something about much less studied ideas in alignment, such as persona alignment. Our understanding of those ideas is much vaguer, and they are much less specified. If we studied them more deeply, we might discover similar deep obstacles, where persona alignment cuts against the grain of coherent agency.

Maybe it is not really possible to have a superintelligence play the Claude persona that some IQ-130 humans came up with, and make the ASI believe that it really is that character. Would a superintelligence really go along with the reasoning in this strange human-written constitution?

Furthermore, people seem to expect Claude to be, among other things, corrigible from its constitution. But as we have seen, corrigibility on its own seems to have deep conflicts with coherent agency. The constitution basically circumvents these conflicts by telling Claude in writing "be corrigible".

The evolutionary analogy also seems to undermine the prospects of persona selection alignment, one of the strategies that Anthropic is using. See Anthropic’s persona selection model.

In persona selection alignment, you essentially want the AI to always play a certain character or persona. The training procedure reinforces the model to play that persona. I do not think that training the AI to play a persona creates a precise inner objective for the model to always play that persona. I also do not think that playing that persona would necessarily be comparable to the “personas” that human beings have.

Humans play characters in some sense because of their own preferences and goals. If you are sad about someone close to you dying, you are, in a sense, playing the persona of a sad person. Similarly, when something joyful happens to you, you could be described as playing the persona of a joyful person.

If we build an AI optimized to play a character like that—playing a sad character when something happens that would make a human being sad, or a cheerful character when something happens that would make a human being cheerful—the drives and preferences we end up with in that model could be very non-human-like. Even if we were somewhat successful in embedding a preference or drive to always play that character, this would be very different from human beings.

The second part of the evolution argument was about building an environment, a civilization, according to your preferences. Human beings, given the option, would minimize things that make them sad and maximize things that make them cheerful. If an AI’s true objective were to play a certain character, and it reflected on that objective, what kind of world would it build? Would it be the same as the world a human being would build—someone who was not optimized to play that character, but who really did experience sadness when something bad happened to them?

If the AI instead learns heuristics and proxies to optimize for that were in some sense related to playing a character, what kind of world would it build if it had that option and reflected on it?

Can we just automate alignment?

Since Plan C is trying to solve alignment in just a couple of months, it can only work if we get some large win from automating alignment research with AI.

Many people in the AI development community seem totally convinced that we are just going to automate alignment. (I am caricaturing a little bit here.) According to them, we do not really have to struggle with alignment at all. We just have to spend X% of our compute on alignment for a couple of months. Until the point comes where AI can align AI, we do not really have to think about this too much.

“we're going to hold back for three months, get this right. We're going to dedicate 90% of our compute to automated alignment research”

“we're going to hold back for three months, get this right. We're going to dedicate 90% of our compute to automated alignment research”

The fact that this is the main plan for alignment should make us pretty pessimistic by itself, just because of how obviously dangerous this sounds.

One problem with answering whether we can just automate alignment is that there have not been many serious proposals explaining how this would work or what we would be optimizing for. If we do this naively and simply have the AI come up with alignment plans, we are obviously optimizing it to produce plans that are maximally convincing to the person reading them.

The default outcome seems to be a last-minute attempt to vibe-code an implementation of an extremely flawed idea, but one that sounds amazing to the people reading it.