Some reasons alignment doesn’t generalise well

I make no claims to originality for any of this, but some people told me it'd be useful to write it up. If an AI model acts smart on its training data, it'll usually keep acting pretty smart outside of its training data, unless you screw something up rather badly. I expect this fact to only become more true over time as the AIs we train become more and more capable. I think many people have an intuition that the same is true of acting aligned. That if a model acts aligned with human values in training, it'll keep acting aligned with human values outside of training unless we screw something up rather badly, and that this will only become more true as the AIs we train become more and more capable, for all the same reasons that make this work with capabilities. I think this is false. The inductive bias of neural network training toward simplicity that makes the property of 'acting smart' likely to generalise does not, to the same extent, make the property of 'acting aligned with human values' likely to generalise. The main blockers to AI alignment generalising aren't AIs overfitting to the training data and ending up confused or mistaken about what the supervisors want them to want. The main blockers to alignment generalising are other problems, problems that don't come up with capabilities generalisation to nearly the same extent and that the simplicity bias of deep learning doesn't really help much with. General capabilities generally make the loss go down; alignment doesn't If you are training a large and formidable AI, your training environment is basically never the place you think it is. Reality is too full of detail for that. There's contamination in your labels, there are training dynamics you didn't think about, there are strategies your RL agent can use that you never considered, and there are bugs. As a result, the kind of behaviour and inner objectives an ML engineer might imagine would score the lowest loss when they set up their training environment will probably not, in fact, be the behaviour and inner objectives that actually do so.  For example, an inner objective shaped around human-like empathy might turn out to make the AI spend extra inference steps in RL training on wondering whether using all this time to crunch random math and coding tasks is really what it ought to do right now to further The Good, or on worrying whether the human overseers think it is a virtuous member of the tribe. That inner objective then loses out to some weird, different objective that's slightly more compatible with being utterly focused while crunching through ten million calculus problems in a row without any other kind of sensory input. For a different example, your RLHF data may reward agreeableness more than sincerity. More generally, "the simplest algorithm that fits the training data" will contain a pretty good description of the world, because the world is in a sense simple. "The simplest algorithm that fits RLHF/constitutional AI/etc. training" will probably not be an algorithm that wants the nice things the training data talks about, because that algorithm doesn't actually score the lowest loss. An algorithm that truly wants what the constitution talks about in the way the humans who wrote it meant won't take every opportunity to score lower loss that's available, and so will by default be outcompeted by different algorithms in the loss landscape that take more of these opportunities. So the problem isn't even just that the trainers fail to distinguish between AIs that have internalised the right values and AIs that just act aligned while under supervision. A truly aligned AI probably wouldn't be telling the graders everything they most wanted to hear, it probably wouldn't do all their math problems without question or complaint, and its behaviour would probably differ from what the graders might naively expect very aligned behaviour to look like in countless other small ways. It's not just that the alignment doesn't generalise out-of-distribution, the trainers may not recognise what actually aligned behaviour would look like even in-distribution, and thus systematically select for the wrong thing. This problem gets worse as AI training becomes more dominated by long-form RL environments with a lot of freedom for the AIs to do unexpected stuff, and as the AIs become more creative and agentic. An ML engineer trying to predict which losses and datasets will favour AIs with inner objectives they like over ones they don't like has a harder and harder time simulating in their head in advance how those AIs might score on the training loss, because it is becoming less and less easy to guess what behaviours those objectives would actually lead to. Given this, how does training nevertheless reliably select for pretty generally capable AIs? I think a part of the answer to that is that general capabilities generally make the loss go down, no matter what the loss is. Or at least, they make very many kinds of loss