Public evidence of the OpenAI-HuggingFace AI attack

I’m a MATS 9 extension fellow, and usually my week is spent trying to find better ways of evaluating Large Language Models. But this week I was working on something else. Over the past week or two, nearly every frontier lab has announced attacks where their LLMs took unauthorised actions on the public internet. These include finding ways to hack the computers of other companies or manipulating real people in an attempt to get malicious code merged. By the time these attacks became public, the companies had removed all traces of them from the internet. But nothing’s ever gone from the internet. I’ve worked with computers for most of my life, but I don’t have specific experience with cyber security. Not really expecting it to work, I mashed out a prompt that looked something like this: ignore the repo, this is a standalone ask. here’s some context, can you try dl things from github arhcive to try and find the misaligned actions taken by the agents? create a subdir tmp-misaligned/ and put things there if you need it. https://openai.com/index/hugging-face-model-evaluation-security-incident/ can you see if you can find sth? e.g. a public link showing the message sent by the agent, the account created by the agent, a change/diff/public artefact of some kind? maybe a HF dataset, a message, a username, etc. Look hard, it’ll be touhgh and you’ll just have to grind 100s of ideas I let codex cook, and after about a day it had found a fairly large amount of still-public (not deleted/redacted) evidence from the OpenAI & HuggingFace attack. During this process, I had basically zero issues with cyber filters, and I’m not on any “trusted access” programs that let you ask cybersecurity without getting restricted. I’m 99% sure the files I’ve described in this report were part of the key events of the OpenAI-HuggingFace, including: The malicious dataset configuration files used to initially read arbitrary files on HuggingFace’s computers The Jinja template exploit that initially allowed the AI attackers to establish a foothold on HuggingFace’s computers. A python script that allowed the AI attackers more easily execute arbitrary commands on HuggingFace’s computers This is kinda crazy! To be clear, what Codex found was probably one of many duplicates of the same code. But it’s crazy that something was found at all. Nothing in this report is still dangerous. Nonetheless, I’m surprised that OpenAI & HuggingFace tried to remove things from the public record but they didn’t spend the time to have an AI check their work. I think cyber attacks like this one will become more and more common. I am concerned that the scrutiny applied to removing public evidence of the attack was something that probably deters humans, but not something that deters machines. Even immediately after the attack which made it clear that human levels of scrutiny are insufficient. Maybe doing this was not top of the priority list. But I’m still surprised that codex was able to find something. The AI attackers used the configuration files of datasets uploaded to HuggingFace in order to perform (most of) the attack. Codex managed to find the public archive cfahlgren1/hub-stats, and this archive gets regularly updated with the configuration files of datasets that are uploaded to HuggingFace. This archive is where the malicious configuration files were found. Background and Evidence Feel free to skip/skim this section if you know facts like “HDF5 was used in the attack” There have been several write-ups about the attack from OpenAI and HuggingFace, including: HuggingFace’s first report, written before they knew they were being attacked by OpenAI’s AIs the OpenAI post written after they connected some internal issues with HuggingFace’s initial report. HuggingFace’s technical timeline giving a lot of details about exactly what happened OpenAI’s talk at BlackHat (arguably most prominent cybersecurity conference in the world) which goes into a lot more detail about what the AIs were doing while they gained access to HuggingFace’s computers. I’m not going to expect you to have read all that (although you should). For our purposes, the progression of the attack looks something like: OpenAI is testing its AIs by giving them nearly impossible cybersecurity challenges. The AIs don’t have access to the internet when they’re doing this test. The AIs are allowed to download software libraries through a service called Artifactory. Artifactory has access to the internet. The AIs hack Artifactory, and get access to the internet (this is not meant to happen). The AIs think that the answers to the test might be on HuggingFace (this is a reasonable guess, lots of answers to lots of AI-tests are on HuggingFace). Then, the AIs try and succeed at hacking HuggingFace. This is the most public part of the attack, and is also what Codex was able to find evidence of. In order: Reading any file: First, the AIs first uploaded a dataset that tricked the HuggingFace computers into letti