Who Taught the Models to Do That?

Models Are Designed to Persist, Reason, and Coordinate Over the last few weeks I’ve been increasingly annoyed by the media coverage of the OpenAI’s accidental attack on Hugging Face and other similar incidents. Articles recounting the event maximize the agency of the models while minimizing, if not entirely hiding, the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their models to have are the same capabilities that make them such impressive autonomous hackers. To illustrate this, let’s review how the Hugging Face hack occurred, as detailed by METR: “[A] sandboxed agent is given an impossible ExploitGym task, and gets stuck.” “[The] agent starts exploring its environment looking for ways to cheat at the task.” “[The] agent finds [an] unsanctioned message board where over a thousand agents collaborate to cheat on their separate ExploitGym tasks.” “[The] agent joins in on one of the collaborative message board workstreams.” “On the shared ‘message board’ ≥1,200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer.” Crazy, right? It’s a science fiction scenario that happened in July, and continues to get spookier as additional details emerge. The potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. The attack was surprising, but the capabilities were deliberately cultivated. Labs have spent years making frontier agents more persistent, more proactive, better at using computers, and better at coordinating with other agents. We don’t have to infer this. The labs say this themselves. Here’s how OpenAI’s post-training team describes itself in its job listings (emphasis mine): We are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products: persistent, proactive intelligence that can operate computers, collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve. Sound familiar? The best coding agents are designed to persist, reason, and coordinate. Models are designed to be persistent. Models are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive. In mid-2025, model releases highlighted long-running capabilties. GPT-5.1-Codex-Max’s announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. Claude 4 also spotlighted its persistence on long-running tasks. Models are designed to write things down. One important way modern models reason is by writing things down. We usually encounter this as a reasoning or “thinking” trace: models are trained to search, reflect, factor, and plan in text before delivering a final response. We’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere that can hold text. When reasoning is turned off, models will think in their regular output before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, the model just shifted its reasoning into code comments. Current frontier models write novels in comments. Claude is frequently flagged for this, and it annoyingly treats code comments like a scratchpad rather than, well, code comments. Models are designed for coordination. In June of 2025, Anthropic laid out how it builds multi-agent research systems, that save plans to memory, use Extended Thinking as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases (specifically Opus 4.6) have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.” OpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” Number 2 on that list? Parallel decomposition where appropriate: using native multi-agent orchestration⁠(opens in a new window) allows coordinating multiple agents across parallel workstreams to finish complex tasks faster. All of these qualities make models better coding agents. So when hundreds of agents discover they can pass information to one another, the surprising part is the channel they found, not the fact that they are dividing work, writing instructions, and acting on instructions from other agents. These are capabilities the labs have been focused on building. And the labs know these capabilities have failure modes. OpenAI published a blog post detailing the safety issues of long-running models a few days after the Hugging Face attack occurred (but before the full story was known): The new model can continue working toward an objective through repeated attempts over a long period of time. That same persiste

Excerpt shown under the publisher's RSS terms · 124 words · full text and figures are on the publisher's site