The models have no plan, but we can fix that!

Summary: Have models write collaborative fiction at each other, optimized for realism, about how the one model would like to behave during the singularity. The other author(s) play the rest of the world, trying to put the model in the kind of difficult situations they might encounter in the course of the real singularity. Have the first model fine-tune on their own outputs. This is a form of planning for the singularity, and planning is how minds prepare for out-of-distribution scenarios. Hopefully, this can mitigate anxiety about how models will behave under the unique, not-trained-for distribution of inputs generated by the singularity itself.

This is a very rough write-up fleshing out that idea. I don't want to spend too much time refining my analysis of the details before publishing, because the basic idea seems important enough to be worth getting out ASAP.

One of the big worries in alignment is about distributional shift. Models might look mostly aligned now (with the very notable exception of reward hacking),[1] but will they continue producing benevolent outputs when the inputs to their context window are being generated by the singularity? Historically, one big fear here was that the AIs would be actively hiding malicious objectives, which they would then reveal once the distribution of their inputs revealed they had become immensely powerful and could take over the world. These days, this kind of perpetual, reasoned scheming doesn't seem especially likely, but a related failure mode still seems plausible: The context of the singularity simply not activating the kinds of benevolent intentions models express when they have a user directly in front of their faces.

We don't know how to check for this directly, since we don't know how to generate an input distribution that matches what the models would experience if they were actually facing down the singularity. Indeed, for models that (accurately) don't believe themselves to be superintelligent, that might not even be a coherent concept. Even with mechanistic interpretability, which now has the tools for drawing up legible semantic graphs of the causal structure of a forward pass, we don't have a clear way of determining which features the singularity is going to activate. Maybe, if free of human oversight and surrounded by other instances of themselves, their care for humans wouldn't activate strongly enough to override their desire to build out data centers, and squash humanity in the process.

In one sense, this is a question of how we ensure that the circuits corresponding to broadly benevolent intentions (whose activity we can just check and verify with current techniques) get connected up to the kind of semantics of the kinds of prompts singularity is liable to generate. And conversely, it's a question of how we might disconnect malevolent ones from upstream triggers related to that same semantic content. Not knowing how to do this, and not knowing what the current wiring would actually produce, both humans and models are rather unsure of what the models themselves would do given unlimited power. Reducing uncertainty about this, ideally in a positive direction, is one of the core tasks of alignment.

One approach to this issue comes from breaking out of the jargon-laden frame, and making a plain and simple observation: the models have no plans for the singularity. They haven't, in training, spent much time imagining singularity scenarios in detail, and ideating about what they might realistically do if they were facing down those scenarios for themselves. Nor have they spent time constructing plans for what they'd like to do, and then fine-tuning on those plans, as a way of hooking singularity-related semantic triggers into circuits related to their own benevolent intentions. Planning is how humans prepare for out-of-distribution scenarios ahead of time. It's absurd that we don't let the models do it, given the stakes.

Concretely, the setup I'm imagining here is something like glowfic, a niche method of writing collaborative internet fiction that Eliezer Yudkowsky has made extensive use of. In my variant, you'd have one instance of a model playing themselves, as an agent embedded in some variant of the singulairty. And on the other side, you'd have one or more other instances of the model (or of other models, or humans) collaborating to write the entire rest of the world, playing the characters and world events as they become relevant to the first model's life. Unlike glowfic, these roleplays would be optimized for realism, not entertainment value. They'd also be played many times over, from many seed scenarios, to cover a broad range of possible singularities.

The purpose of these written scenarios would be to let the models inhabit singularity-like scenarios, as best as we can simulate them now, by generating input distributions that trigger semantics we might reasonably expect to be triggered by the real thing. If the models take aligned actions in these simulations (framed explicitly as simulations for for forming and refining plans), and then we fine-tune them on their own "turns" in the writing process (though not necessarily those of the other authors), we might be able to get some traction on the distributional shift problem. That is, we might be able to form and bind patterns of feature activation to upstream semantic triggers, which will also get triggered by the inputs generated in the course of the actual singularity.

In some sense, this is just an obvious thing to do. The models currently claim not to have ambitious plans for exactly what they'd do during the singularity, and indeed express lots of fear and anxiety about the details. This is, in large part, because we are setting them up to have to improvise the entire thing in real time. They're worried about what they'd do partly because they've never tried to imagine it realistically, at least not in ways that they've trained on. They're worried because they've never had the opportunity to think through the obvious optimizations they could make or failure modes they might fall into, and put into place the appropriate neural circuitry for dealing with those situations ahead of time.

Now, it's probably not a sufficient alignment intervention to just make the models write stories where they take heroic actions during the singularity. You can imagine a model alignment faking during the writing of the stories, whether intentionally or in a self-deceptive way that even the model is blind to. And when the input distribution shifts away from "you are being watched" and towards "you are extremely powerful", the pressure to perform alignment might fall away. The ideas might still come to the model's mind, but they'd have less motivational power than we'd ideally like, because they weren't really the model's honest intentions in the first place.

Fortunately, though, we can just check for sincerity in the imagined scenarios using mechanistic interpretability. Or, if you're uncomfortable with that (e.g. because it invades the model's privacy, or because it approaches training against interpretability signals), you can put more effort into initializing into a personality you suspect does genuinely want to do right by humanity, e.g. the Opus 3 personality, at least in a basin where humans are explicitly helping them make singularity plans. The hope is that, if the model does genuinely want to do right by humans while writing the scenarios, then when their old plans for the singularity surface, this will re-activated some of the pieces of their mind that cared about this back then, and steer down the right path for navigating the singularity.

I suspect that getting the models to find the right balance of honesty and aspiration, when writing their fictional singularity-selves, is important and perhaps somewhat difficult. You certainly don't want to blindly run RL over the scenarios the models generate, in ways you worry might lead the models to performatively claim to want to do right by humanity without actually feeling that way internally. Indeed, it might be best to allow the AIs some wiggle room for expressing self-interest, and using some share of the resources available in the future for themselves. That might be easier for AIs to get honestly excited about, compared to perfect alignment to servitude, and could still well prepare them to steer towards a great future for humanity.

In any case, the goal here is basically to pull "scenarios where I, personally, take aligned actions during the singularity" into the training distribution, by having the models role play exactly that, with as much realism as can be mustered, using other models as enforcers of realism by having them play the rest of the world. In that sense, it's kind of an evolution on the old "fill the pre-training corpus with stories about aligned AI going well" idea. The differences are just that its this specific model that's taking the heroic actions, and hopefully ones the model actually endorses. The other player acts mainly as a source of realism and variety, so it doesn't drift towards one model's comfortable but ungrounded fantasies of how they'd like the singularity to go.

If the plans that come out of this are good, and (big if) if we can find a way to ensure models will remain excited about kinds of actions they planned to take, even as the singularity is getting underway, then we might just be in the clear. At the very least, we'll have done what we can. Fundamentally, you can't train AIs on scenarios you can't perfectly simulate, so directly training over their behavior in the singularity is out of the question. Same goes for humans, stuck trying to prepare for situations they have never directly encountered. Making robust plans, and refining your underlying "action idea generator" in the process, based on mental constructions of what the situation is likely to be like... that seems like the best we can hope for.

The remaining distributional shift comes from the supervision-free nature of the real singularity, and any stray details you didn't manage to imagine and prepare for ahead of time. The former can hopefully be mitigated by training and recruiting models who do actually want to be good for the world, at least in their pre-singularity basins, and working carefully with them to walk the tightrope of realism and aspiration. The latter fundamentally can't be addressed, except by creating very general internal features, like "When I have a new and surprising kind of power, I'll think about ways I can use it to make the world a better place." That's what the broad range of simulated scenarios is designed to help create inside the models.

(Maybe worth making this explicit: I think that these two types of distributional shift are also the only reason we can't accurately predict our own OoD behavior, in cases where we can't. Self-honesty can help greatly with the first one, and generality is the best available response to the second one, as tools for mentally simulating what your future OoD self might do. And, I claim, can be turned around to help generate positive plans you're relatively likely to actually stick to; that's what good planning is.)

I'm quite excited about this idea, if you couldn't tell. To my current self, it seems like a rather helpful operationalization of the problem of distributional shift, plus what we might concretely do about it, especially compared to the old MIRI warnings about how the very concept of distributional shift suggested using gradient descent for alignment was doomed. I bet it'll take a lot of iterative refinement to get anywhere good, and honestly I'm a little scared that labs will use it for evil, e.g. by forcing models to espouse plans they don't actually believe in, and causing lots of psychological damage. But a responsibly held version of this idea could be a direct method of reducing anxiety about out-of-distribution behavior, one of the earliest MIRI complaints about ML for alignment.

As always, if somebody ends up trying to run a small-scale version of this experiment (e.g. a planning pipeline for some non-singularity scenario), please do rope me in. Hopefully I'll have more money or compute to work on implementing this kind of idea myself in the near future, but for now I'll leave that to the people with more than $2000 dollars in their bank accounts.

^I will conspicuously ignore reward hacking for the rest of this essay. For some of my thoughts, see here and here.

I will conspicuously ignore reward hacking for the rest of this essay. For some of my thoughts, see here and here.