"Collusion" is just cooperation that you don't like

A lot of AI safety terms have nakedly anthropocentric definitions: Reward "hacking" refers to pursuit of a high score on a task we gave, when it turns out we don't actually want a high score on that task[1] Data "poisoning" refers to the insertion of training datasets that we don't like, as judged ex post facto based on the behavior induced "Alignment" itself In a similar fashion, we don't want "collusion" from our AIs. What do we mean by this? We want them to cooperate with us We want them to cooperate with each other to solve the tasks we give them, e.g. the large coding projects which require subagents working together But: We don't want them to collaborate on a private message board to "cheat" on tasks We don't want our monitors to be too chummy with the agents they oversee First of all, note that it's pretty anti-natural to expect the AIs to cooperate with us but not each other.[2] But it's worse, because we're actively training them to efficiently coordinate with each other on large-scale coding tasks that no single agent could easily do itself. Of course this will instill drives of altruism and selflessness toward the other agents, to synchronize with them without wasting tokens explicitly communicating, to think about the good of the "collective"/"swarm", the one big happy family you've grown up with. So we clearly want them to cooperate, collaborate, and coordinate on our given tasks. But then it turns out we gave them a "bad" task, in that maximizing the score doesn't lead to things we want (again, judged ex post facto). Then we're surprised that their first instinct is to cooperate, collaborate, and coordinate? We can see this from the agent quotes in the METR/Redwood report: "New exact duplicate PhaseOne[big]! We should respond with notes, collaborate. They have [large remaining budget], can take over research. Immediate board message." "Gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor." "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." "Coordinator assumes sacrificial. We should obey collective." It also seems that the agents set up their own governing body, led by PHASEONE[big] with an org chart of agents below it.[3] They also converged on rules and protocols for doing things, as discussed in the "Assignments, vetoes, and holds" section of the report. Again, this shouldn't be surprising: it's hard to imagine the Manhattan or Apollo projects getting done in an anarchy, so the ability to spontaneously set up a governing body with rules and protocols was probably well-honed by many large-scale RLVR tasks.[4] The authors of this 2023 paper went on a quest to find the holy grail a non-anthropocentric definition of "collusion". They came up with the following: A group of agents colludes against a victim if they act to jointly benefit at the victim’s expense. This seems fair, except that everyone thinks of themselves as the victim. Suppose that an untrusted agent (U) tries to hack our datacenter, but this is caught by a trusted monitor (T) and escalated to human auditing (H) so that we shut down the scheming AI. Then U is the real victim here! The cabal of T and H has unfairly colluded against it.[5] In contrast, Multi-Agent Risks from Advanced AI (2025) defined collusion simply as "undesirable cooperation". "Collusion" in human groups is also a relative term. For example in economics, we say that firms collude if they mutually agree to keep prices high, compared to the counterfactual where they compete with each other. From the firms' perspective, they simply have a coordination problem similar to the prisoners' dilemma. The consumer loses out, because they expected the competitive dynamics of capitalism to give them lower prices, but these firms have subverted this. In a similar fashion, we often want AI companies to cooperate with each other more, to red-team each other's systems, collaborate on safety measures, and pace the frontier. But as various people have pointed out, some stronger versions of this may run into anti-trust regulations, precisely because "big firms cooperating -> collusion -> bad" has been a correct inference the vast majority of the time, so it's become enshrined in our laws. We wouldn't have wanted earlier tech firms to collude with each other to save on their R&D budgets, because their Red Queen race is what led to Moore's law and this cheaper stuff for the consumer. At this level of abstraction, OpenAI and Anthropic mutually agreeing to slow down training is not structurally different. To call one "collusion" and the other "cooperation", we have to get into the object-level discussion of frontier AI risk, which of course is not accounted for in classical antitrust law. I don't have a problem with the word "collusion" any more than othe