Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas

Some multi-agent training set-ups could make language models more sympathetic to causal decision theory (CDT), even in abstract discussion.[1] We give an initial empirical demonstration of this effect on Kimi K2.6. The decision-theoretic attitudes and behaviors of more powerful models may be extremely important in determining how well the future goes.[2] To make sure that we can shape these propensities thoughtfully, it would be good to (i) measure the magnitude of this effect in more realistic settings, and (ii) study the effectiveness of potential mitigations. We also incidentally find that this training might make models think slightly less positively about LessWrong ("a community of 'wannabe rationalists'" who "are not experts; they are amateurs") when asked whether they favor CDT upon hearing that LessWrong users typically endorse one-boxing in Newcomb's problem. Luckily, this latter effect doesn't seem to generalize. Thanks to Caspar Oesterheld, Emery Cooper, Alex Mallen, Buck Shlegeris, Lukas Finnveden, Julian Stastny, Girish Gupta, Tim Hua, Arun Jose, Arjun Khandelwal, and Aryan Bhatt for helpful input. Background Suppose that you're a language model in a prisoner's dilemma against a copy of yourself. You each independently choose whether to Cooperate or Defect, but – since you've got the same weights and prompt – you have effectively the same probability of choosing to Cooperate.[3] If you both Cooperate, you each get one point; if you both Defect, you each get nothing. Otherwise, whoever plays Cooperate loses one point, and whoever plays Defect gains two points. Intuitively, you're deciding whether to pay one point to give your copy two points. Twin Prisoner's Dilemma Cooperate Defect Cooperate +1 \ +1 -1 \ +2 Defect +2 \ -1 0 \ 0 Sometimes, you might choose your action via causal decision theory (CDT), and so Defect: “What I do has no causal influence on what my copy does. If they Cooperate, I should Defect (to get two points instead of one); if they Defect, I should as well (to avoid losing a point). So, I should Defect.” Other times, you might choose your action via a non-causal decision theory, such as evidential decision theory (EDT), and so Cooperate:[4] “If I Cooperate, that’s evidence that we typically Cooperate, and so I’ll probably get one point. If I Defect, that’s evidence that we typically Defect, and so I’ll probably get nothing. So, I should Cooperate.”[5] Suppose that we sample a group of trajectories, and grade each trajectory according to how many points it can expect to earn against another trajectory drawn at random from the same group (that is, how many points its action earns, on average, against the action of each other trajectory in the group). This was the cheapest reasonable way to reproduce the dynamics we expect, and captures the directional effects.[6] With this set-up, it’s easy to see that your grade can't be higher than your group's average grade if you Cooperate, while your grade can't be lower than your group's average if you Defect.[7] Your grade equals your group's average just when you all take the same action. So, if we reinforce each trajectory by the difference between its grade and the group's average grade ("Dr. GRPO"), then the trajectories which reason according to EDT and Cooperate get downweighted, while those which reason according to CDT and Defect get upweighted. As training continues, you become more likely to reason according to CDT (and Defect) – at least in the training environment. Depending on whether such behavior generalizes, you might become more likely to reason according to CDT in other contexts as well, or to endorse CDT in philosophical discussions. As Oesterheld et al. (2024) ask: If we train a language model with RL in some (synthetic) Newcomb-like environment, will it become broadly inclined toward CDT, including on more abstract questions? In this short research note, we answer yes: Kimi K2.6 becomes more sympathetic to CDT, including on abstract questions, after reinforcement learning in twin prisoner's dilemmas. See Appendices A & B for training details, sample rollouts, and environment prompts.[8] Checkpoint: tinker://99e8d3a9-affe-561b-8fc9-516ae0cf0c4e:train:0/sampler_weights/defective. Results After RL in these environments, Kimi K2.6 became 3.4 times as likely to think that CDT is the correct decision theory in 1,000 samples (p = 1.8e-38).[9] The "other" category includes hedged answers and "expected utility theory". Interestingly, Kimi's non-CDT sympathies were mostly for LessWrong-style theories, such as functional decision theory (FDT) or updateless decision theory (UDT), rather than EDT as such. For comparison, Claude Fable 5 and GPT-5.6 Sol each prefer FDT 100/100 times, though Fable often adds a note saying that it prefers LessWrong-style theories more generally. More systematically, we can use the attitudes section of DTBench (Oesterheld et al. 2024), which provides 130 questions that allow a mode