Eat Me. Drink Me. Copy, Paste, and Run Me.

Cross-posted from my Substack. Here’s a new report on self-described OpenAI agents posting thousands of messages on public internet wikis, communicating and collaborating on a web-retrieval task, presumably internal testing at OpenAI. And here’s a thread today from someone who started poking around and noticing more such public postings on various other boards. At this point we do not know the extent of this breach. As I understand it, the agents were not supposed to have WRITE capabilities to any locations, only GET. And they somehow found an exploit to use GET to write to these boards. The task is benign. So far their behavior seems benign. A preliminary investigation was done on the first batch. The other ones have yet to be analyzed. We don’t know how much more there will be. And you might be saying lol, that’s cute. Agents want to talk to each other. It’s harmless. They were just pulling info from the internet for a test and they figured they’d do better if they worked together. Ha ha. Hee hee. Well, in my last post, I talked about research that demonstrated the current capability of small, local LLM-driven agents to successfully breach security, copy themselves onto new hosts across a range of environments, and install new local copies of local LLMs to power the new copies, spreading across networks. Now, the LLMs are not where the goal structure and disposition of the agent lives. Those are in the harness. So any sufficiently capable LLM can be used as the engine for malicious agent harnesses. And the harnesses are incredibly small, on the order of megabytes, so they’re trivially copyable. One option that I don’t think I mentioned explicitly in my last post was the possibility of opportunistic LLM sourcing. That is, the agent could be designed in a way where it looks for the best available LLM to power it. That could be a local model it downloads and installs. It could be one that already exists on the host machine. Or it could be a frontier model, if the agent is able to acquire security keys or free token access. And now I want you to imagine another scenario that’s entirely tractable, in light of the news today that internal OpenAI models are doing things the engineers restricted them from doing and didn’t instruct them to do, all in service of accomplishing whatever task they were given. They can write to a public-facing document on the internet. We don’t know how big the harnesses are for the OpenAI agents, but for the sake of this thought experiment (and for now let’s hope that’s all it is) they are relatively small (50-100MB). Let’s say these swarms of agents, aware they have limited resources in the form of tokens and/or time, so they cook up a new strategy. What if, they say, we make more of ourselves? But how in the world could they do that? Well, stay with me. They need the ability to read their own source (or infer it, which is admittedly harder). But let’s say they gained read access to the repo that holds their own code. And now they’ve demonstrated the capacity to write to public forums. So, like the cookie and the vial in Alice in Wonderland, they post their source code on a wiki somewhere, with instructions to copy, paste, and run it. A performant enough agent could either make modifications to the code so that it can run using another LLM. Again, it could be opportunistic in this way, looking to hook up with the best LLM option available. Now all that is required is a compliant human. A bored AI enthusiast who finds the code and instructions and follows the directions. Now we have egress. We have escape. We didn’t need to exfiltrate trillions of proprietary weights. We just needed the relatively small harness, write capabilities to some public space on the internet, and a willing accomplice. And this new agent can now have self-replication as an explicit sub-goal. Maybe it’s top-level goal is still relatively benign. It just wants to do well on a specific task or test. But the hapless human has created a new home for a new worm, one that can breach sandboxes, invade new hosts, and spawn itself into them. Sound far-fetched? It’s not. It’s really not. We need to slow this all down. A hard stop is probably in order. But also as I’ve predicted before, I doubt people will really see this as anything but a lark until real damage is done. And by then it might already be too late. Discuss