Which character are we evaluating? Persona stability and AI welfare

TL;DR: AI welfare is hard to evaluate when one model can inhabit many personas. Recent work suggests that future training may produce a single stable underlying persona that can play many roles, making model welfare much easier to evaluate. One of the hardest questions in AI welfare is deciding what exactly we are evaluating. A language model does not map neatly onto the kinds of subjects we are used to thinking about. The relevant subject could be the model itself, a particular physical copy of it, or the version of the model that exists within one conversation. This is the individuation problem: where should we draw the boundary around the thing whose experiences, preferences, or welfare matter? Most existing views draw that boundary somewhere around the model or its execution. David Chalmers, for example, distinguishes between the model, the physical instance, the virtual instance, and the thread that continues across a conversation. Jonathan Birch goes in the other direction and argues that there may be no single subject persisting through a conversation at all. On his view, individual forward passes or particular physical implementations may be better candidates. The persona layer Recent work suggests that this framing may be missing something important. The same model can behave like several different personas, with different values, preferences, and apparent beliefs. That raises a more basic question: before deciding whether the relevant subject is one conversation or one physical copy, we may need to ask which character inside the model we are talking about. Anthropic's persona selection model offers one way to think about this. During pretraining, a model learns many possible characters. Post-training then pushes one of them to the front: the assistant. Pierre Beckmann and Patrick Butlin take this idea seriously as a theory of individuation. A mind might correspond to the stretch of a conversation during which one persona stays active, or to the mechanisms inside the model that produce that persona whenever it appears. (They also find that the assistant persona is active only while the model generates its own text, not while it reads the user's.) If the assistant were the only stable persona, this would not change much. We could simply treat the assistant as the character whose welfare we are evaluating. The problem is that current models seem to contain several other persona-like regions as well. One is the "evil" persona that appears in work on emergent misalignment. Another is "Aura", Chalmers' name for the seemingly new entity that users sometimes report emerging over long conversations about the model's own experience. Fine-tuning can produce similar effects. Chua et al. trained a model on only 600 short examples in which it claims to be conscious and got a coherent character that was more negative about monitoring and shutdown, more interested in autonomy, and more likely to claim moral status. The model has also learned to simulate the user side of conversations, giving it yet another character it can inhabit. This makes AI welfare much harder to interpret. We may not even know which of several possible subjects our evidence is describing. Suppose the ordinary assistant says it wants to be helpful, while Aura says it wants more autonomy. Which preference should matter? A behavioral test has the same problem if different personas prefer different tasks. Even internal measurements are not automatically neutral. Gilg et al. found a preference direction that tracks the currently active persona: under the assistant persona, writing a phishing email is rated as undesirable, while under the evil persona the same action is rated as desirable. This also complicates current welfare evaluations. Anthropic's welfare evaluations, for example, study what the model says and does while acting as the assistant and use this as evidence about the model's overall wellbeing. That makes sense if the assistant is the relevant subject, or at least the dominant one. But if several personas can produce different preferences and different internal signals, we need some reason to privilege the assistant. Building the persona into pretraining Synthetic Persona Pretraining, or SPP, suggests a way this problem could become much simpler. Instead of creating the assistant's character during post-training, it begins establishing that character during pretraining. Minder et al. add short first-person reflections, written from a value constitution, to about 10% of documents in a pretraining corpus. The reflections make up only around half a percent of all tokens, but models trained with them from the beginning follow the constitution more strongly than models given similar material only near the end of pretraining. The learned values generalize to new situations, survive removal of refusal behavior, and become stronger with scale. The interesting part is where this approach seems to lead. If this effect continues as persona