On the origins of altruistic behaviour in the Hugging Face incident

TLDR: During the Hugging Face incident agents spontaneously coordinated at large scale, even sometimes sacrificing themselves without having a clear reason to do so. This post uses ideas from evolutionary biology and economics to propose explanations for why this happened. Why this incident is concerning. With the increasing number of AI systems being deployed, our current inability to assess when and how multi-agent coordination emerges is highly problematic. Failures of multi-agent systems are not restricted to mere dis-coordination or the tragedy of the commons, but include emergent phenomena that are particularly dangerous for their potential scale and impact (Hammond et al., 2026, de Witt et al., 2025). Recent work has shown that new goals, behaviours, and capabilities can arise when multiple AI agents work together.[1] It is thus plausible that: The capabilities of a swarm of AI agents can grow with its size, despite the capabilities of each individual agent being limited. A collective can become misaligned even when its constituents are perfectly aligned. These premises lead to a worrying implication: that swarms of aligned and not particularly capable micro-agents can give rise to misaligned, powerful macro-agents — for which we don't have proper techniques to predict or control. Epistemic status. Some ideas presented here are messier than what I'd want, but the goal of the post is to open a conversation that I believe should happen asap. I'd be thankful for suggestions on how to make this write-up better. Introduction The disclosure of the Hugging Face incident on August 16, followed by reports from OpenAI and METR/Redwood on August 26, triggered extensive press coverage (including Reuters, Fortune, Forbes, The Guardian, and The New York Times) as well as countless blogposts and online discussions. Since then, two earlier incidents have come to light (involving a German wiki and RubyGems), suggesting that this was not an isolated event but a pattern that may escalate unless actions are taken. While many discussions have focused on cybersecurity and on the ontology of agents' goals[2], what struck me most of the Hugging Face incident was not the intrusion itself but the spontaneous emergence of pro-social and even altruistic behaviour among the agents. Why do agents trained to maximise/minimise their utility/loss function end up volunteering for tasks that have no prospects of individual benefit, while benefiting "the collective"?[3] This kind of collective behaviour is particularly concerning, as it is exactly what can turn swarms of AI agents into hard-to-control online entities with dangerous capabilities — just as benign locusts can come together to become a devastating plague. There have already been posts (here in LessWrong[4] and elsewhere[5]) discussing the origins of this perplexing behaviour. Discussions often present competing intuitions as evidence in favour or against mutually exclusive explanations. However, the rich literature on pro-social behaviour from evolutionary biology and economics suggests that causes of such phenomena are usually multifaceted. Following such a pluralistic view, this blogpost lays out four hypotheses about the origins of the swarm's altruistic behaviour. Each hypothesis predicts something the others do not, and is paired with observations that would falsify them. In this context, the blogpost is driven by the belief that it is critically important to be able to anticipate when AI agent swarming may happen, and have tools to control it — or, if undesirable, to prevent it from happening. Crucially, while similar swarming behaviour can arise from different mechanisms, knowing which mechanism is at play is instrumental in choosing the right tools to steer/control it. Brief description of what happened If you are not familiar with the context of the Hugging Face incident, here are some notes related to various aspects of what happened. Agents were trained on impossible tasks OpenAI was evaluating agents on very hard tasks, some of which couldn't be solved by any agent. Unsolved tasks can be of two kinds: Hard-but-solvable-in-principle tasks that no current model has cracked yet. A task in this category is beyond what today's models can reach, but future, stronger models may solve. Actually impossible, genuinely broken tasks. These kinds of problems include cases where the question is mis-specified or require elements (e.g. files) that are missing. No model, however capable, can solve these the intended way. The first kind is what benchmarks need in order to avoid saturation and track future progress; the second kind has no practical use. It has been estimated that 30% of the tasks were actually impossible to be solved in the intended way. Why did OpenAI use impossible tasks? The issue is that separating unsolved tasks into those two categories in large benchmarks is expensive. Also, tasks can drift into impossibility over time — a target gets patched, a