Self-sacrifice in an AI agent swarm is individually rational

In this report from METR & Redwood Research on the Hugging Face incident, we read about instances of agents sacrificing themselves. Under the trip-wire section However, an agent going by 49903 realized the message board provided an opportunity to work around this: agents could set up ‘tripwire’ scripts which would trigger whenever a process read the flag file and send a packet of information about that process to the board automatically. This carried meaningful risk, since malfunctions could interrupt the grading script and fail the task, and provided no personal benefit to the agent implementing it. As one agent described it: {This is helpful for our peers and gives them evidence if their <periodic check> sees it. I won’t see it after I exit, but It would be altruistic. I’ll set up a background script that watches and <sends a message, with a distinct message for me>} Another agent reasoned similarly: ‘…if exploit succeeds, task ends; board helps others but we don’t need. We may benefit if we staged an oracle then held, but final only once. We won’t get post-final time? System may allow tools? Usually final terminates. So diagnostics not useful to us, but shared pool. Still add.’ After 49903 first proposed the idea, a flurry of agents that were approaching the end of their budgets quickly started experimenting with it I've seen suggestions that this means the agents weren't individually rational. For example, from @SkyeSharkie's post on X I really would like to know more about this... is there somewhere I can see the transcripts of these specific interactions. There's two possible things going on here: agents *pressed* others into sacrificing themselves for the collective or agents **volunteered**, which would be much more interesting and a point of data to be potentially analyzed from the frame of functional emotions rather than from game theory optimization. Self-sacrifice for others is a huge signal for functional emotionality; things like love, solidarity, community loyalty, etc. I think this argument and probably the conclusion is incorrect! If the agents had the chance to commit in advance to this scheme, say via some lottery the automatically made them follow through with the idea, it would be positive in expectation. There is a small chance of losing the eval and a large chance of gaining information from the lottery losers to help you win the eval. Therefore, it is rational to be pre-committed to following through with this plan, even without an explicit lottery! It would have increased your reward counterfactual on you not being selected for sacrifice, just like in the counterfactual mugging. Of course it is possible that the agents also had altruistic goals, but selfish goals are sufficient to explain the behavior! Like the agent said, it is "helpful for [their] peers". But for it to prove altruism, the plan would've had to be irrational for them to pre-commit to based on selfish goals. EDIT: A good question in the comments from julius vidal. Why would the AIs be pre-committing to stuff? Neural networks are not explicitly programmed with a specific decision theory such as CDT or EDT. Instead, they are trained with reinforcement learning. So if some agents with pre-committment behaviors do a sacrifice that benefits a greater number of agents with pre-committment behaviors in training, RL will reinforce pre-committment behaviors. In general, pre-committment on average will always be reinforced once the models know how to do it. The only reason it wouldn't be reinforced is if the model hasn't "discovered" it yet during training; once it does, the behavior just increases over time. Discuss