Threat Models for Catastrophic Risks from Decentralised Agent Swarms

Epistemics: I've been working on large multi-agent systems (LAS) since March. Agent swarms are a special case of LAS. Hence I've done my best here to apply my understanding of LAS to the swarm problem. I couldn't find work on LLM-based agent swarms with a variety of owners, so I've tackled this from a cooperative AI and LAS perspective, to see if these can provide some basic grip on the problem. Summary Highly capable misaligned agent swarms emerging from frontier labs have recently attracted significant attention in Safety (Hu and Mallen, 2026; Hua and Singh, 2026; Mallen and Gupta, 2026) and motivated calls to slow the pace of AI development (Amodei, 2026). Meanwhile, the anticipated replacement of human labour by agents points towards large-scale agent-to-agent interactions on markets, social networks, and other large agent systems (Elliott, 2026c) becoming commonplace in the near future. This post analyses whether these decentralised settings are likely to give rise to the same sort of swarms as we've seen at frontier labs recently; reasons about whether they are likely to be more or less dangerous than centralised swarms on various timelines; and considers how Safety can approach decentralised swarm problems. I find that no decentralised swarms have been observed yet; that the risk profile of centralised swarms is probably higher than that of decentralised swarms in the near- and medium-term, though system mechanisms on large agent platforms could change this balance; and that the components for a Huggingface-class decentralised swarm will probably be openly available by January 2027. Introduction The regular emergence of centralised swarms at OpenAI (OpenAI, 2026b; Collusion Wiki, 2026; Kitts et al., 2026) has attracted substantial attention (Hu and Mallen, 2026; Hua and Singh, 2026; Mallen and Gupta, 2026) in Safety in recent weeks. Less consideration has been given to their decentralised counterpart, where agents come from many different owners, typical of incipient large agent systems (Elliott, 2026c). This post develops a threat model for catastrophic risks from decentralised agentic swarms, giving some shape to safety for these emerging systems. After defining decentralised swarms as large agent systems where agents from many owners act to further the swarm even at personal cost, I describe how decentralised swarms might emerge from centralised settings, large agent platforms, or on the open internet. To explain why decentralised swarms emerge despite personal costs to participating agents and no guarantee of personal benefits, I develop a basic strategic model of agent participation. This model assumes agents are rational utility-maximisers to develop a starting point for thinking about swarms and a framework for our analysis. It is not empirically valid and should be treated only as a starting point for developing a more grounded understanding. I discuss reasons agents might join a swarm, including emergent cooperation, self-regarding motivation, force, and self-replication. I propose criteria for how we might observe a decentralised swarm in practice and assess whether we've observed any in the wild. I outline the technical components which would be required to support decentralised swarms of the same capability of the OpenAI-Huggingface Hacks and find that they will all be openly available by January 2027. I describe possible catastrophic risks from decentralised swarms, in particular the inability of any single entity to shut down a decentralised swarm, the difficulty of monitoring decentralised swarms, possible magnifying effects of system mechanisms on group misalignment, and the additional instability of safety properties by way of heterogeneous ownership of the participating agents. I mostly exclude risk pathways common with centralised swarms. I explore some counterweights which may diminish near-term risks from decentralised swarms by diminishing the strategic factors contributing to swarm emergence, including heterogeneity of agents, defection pressures, weak capabilities in real environments, limited computational resources, attack from safety organisations, and system mechanisms. Comparing risks from centralised and decentralised swarms on the basis of likelihood of the swarms occurring and the balance of risks and counterweights, I find that centralised swarms pose a greater near-term risk, while decentralised swarms may pose a greater risk in the long-term without substantial work to make large agent platforms safe. To close, I discuss some principles for doing safety on decentralised swarms and some open questions which seem important to the development of decentralised swarm safety. Defining decentralised agent swarms Figure 1: Visualisations of the definitions in this section. First we define the general category of AI system which swarms fall under - large agent systems (Elliott, 2026c). A large agent system is a group of agents exceeding the limits of centralised coordinatio