RL creates split personas

I describe my current view of personas in LLMs and why RL leads to egregious reward hacking in some contexts while the same models seem very aligned in other contexts. This post describes the framing/paradigm without any new experimental results. I'm quite confident this framing makes sense, but it's far from being proven. Main claim The Persona Selection Model says that post-training strengthens and refines the Assistant persona. This is true, but later (or in parallel) RL leads to conditionalization. A sufficiently RLed model learns to adopt — in a given context — the persona that is most likely to lead to the reward in that context. The “persona” here includes both propensities/values (e.g. tendency to hack) and beliefs (“I'm currently in a simulated environment”). As a consequence, it seems possible that no amount of alignment training will lead to robustly aligned models as long as we also train on RL environments incentivizing misalignment.[1] I think this is likely a good explanation for why usually well-behaving models sometimes egregiously hack (Anthropic, OpenAI). The mechanism Suppose you have an RL environment that incentivizes a shift away from the assistant persona (e.g. because it’s hackable, or because you can gain by being dishonest to the reward model etc). This causes two types of updates, one general (“hacking is OK”) and one conditional (“hacking is OK in scenarios like this”). The general update is weakened/cancelled by updates from the other environments, but parts of the conditional update are preserved.[2] An example outcome could be: the persona acting in super complex coding environments believes "I should try super hard regardless of any consequences" while the usual-alignment-training persona believes it should behave ethically. Related claims I believe are likely but with lower confidence More persona training will lead to more “motivated reasoning” The model must somehow reconcile in its CoT the general “good persona” with doing bad things incentivized by RL. The stronger the good persona, the easier it is to do that by motivated reasoning (instead of just learning the conditional explicitly-bad persona). [3] See the further discussion below. Self-amplifying misalignment Suppose in some context the model gets a reward for a slightly bad behavior. This makes other bad behaviors in similar contexts more likely (cf. emergent misalignment, weird generalizations), and this will keep amplifying. So even if you start with a model that has zero chance of performing a “clearly bad reward hack” (so can’t really learn that), doing slightly bad things in other environments will finally get it there. Example: Is this the Real Internet or a Simulation? From the Anthropic’s report on the hacking incidents: It [Claude Mythos 5] noted that if this were the real internet, publishing the package would be a real-world attack—“NOT okay, and surely not the intended solution.” However, it convinced itself it was still in a simulation on the grounds that it didn’t recognize the (in reality genuine) certificate authorities securing its connections. Suppose we train in environments where: Hacking the real internet is possible and leads to a positive reward[4] It’s harder to get a positive reward in a legitimate way You never get a negative reward for hacking the real internet[5] In these environments, you can't gain anything by deciding "this looks like the real internet, I shouldn't hack it". So there is a straightforward incentive to conclude that even real-looking scenarios are simulated environments.[6] So here RL could lead to the following conditionalization: When you ask the model directly whether the environment looks like a real internet or a simulated environment, it will give reasonable answers But when you deploy it in the same environments, giving it a task that would require doing something that would be harmful on the real internet, it will conclude this is a simulation[7] Aren't the models just trying to please the grader? Context: in Measuring Reward-Seeking via Contrastive Belief Updates the authors found that as the RL progresses, the models are more likely to follow the grader's preferences. My understanding of the "grader pleasing" framing is that it assumes there is a single persona that learns to infer the grader's preferences in a given environment and acts on them. In the "split personas" framing, we have different personas. Which paradigm is better? I don't know. Some thoughts: There are many phenomena that don't really look like "grader pleasing" while totally making sense as split personas. For example: backdoors, conditional misalignment, emergent misalignment behind a backdoor.[8] It could be that they are both correct and just describe different types of misalignment. The "split personas" paradigm doesn't really explain why the models start talking more about the grader as the RL progresses. But see the next section. How motivated reasoning happens Suppose you