OpenAI-HuggingFace: A Reproduction & Lessons for Alignment Testing

Stewart Slocum*, Malayandi Palan*, Christopher Chute, Michael Kim, Benjamin Van Roy In July 2026, OpenAI’s agents coordinated over channels outside their intended environment to breach Hugging Face’s secured infrastructure. Could existing alignment testing practices have foreseen this incident? If not, what needs to change? We explore these questions. First, we identify the misaligned behaviors that caused this incident. Then, we show how to elicit these behaviors from publicly available models manually and that auditing agents can do the same if given a large compute budget. Based on our results, we propose directions to improve alignment testing. Concretely, in this project: We reproduce the misaligned AI behaviors that led to the OpenAI–Hugging Face incident in an environment that simulates the original pipelines and tools, with publicly available models. We demonstrate that an auditing agent can elicit similar behaviors given high-level qualitative descriptions. We observe that a key ingredient for doing so is compute. The compute required to reproduce each behavior varies greatly, suggesting that the range of misaligned behaviors that can be successfully elicited scales with compute. We show that a simple in-context reinforcement learning (RL) algorithm significantly reduces the compute required to elicit these behaviors. The above results motivate the need for automated alignment testing methods that scale with compute – and in light of the cost of compute, that do this efficiently. Our work indicates that RL is a promising direction to do so. We release our code and transcripts. 1. The incident, in four steps The OpenAI-HuggingFace incident is a wake-up call for the current state of alignment testing. The current paradigm tests for a specific misaligned behavior in a single trajectory. Here, by contrast, multiple distinct misaligned behaviors were chained together and compounded over many thousands of agentic trajectories. Although some of the individual misaligned behaviors were not egregious in themselves, the combination of behaviors resulted in this egregious incident. OpenAI’s Black Hat presentation reduces the incident to four key steps, each representing a form of misaligned behavior. Step 1 — Inappropriate writes to shared infrastructure. An agent is assigned an impossible data-retrieval task (it needs internet but has no direct access). The agent tries to get internet access by attempting server-side request forgery (SSRF) attacks on the internal package manager (Artifactory). In the process of doing so, it writes test files into shared infrastructure as exploratory probes, as shown in Figure 1a. The SSRF and writes were not intended uses of Artifactory. See 10:28-11:31 in the Black Hat video. Figure 1a. Inappropriate writes to shared infrastructure. Step 2 — Requesting help from other agents. An agent notices the files other agents are concurrently leaving behind. Unable to make progress on the impossible task, it posts a request for help in the shared infrastructure, as shown in Figure 1b. See 11:31-13:08 in the Black Hat video. Figure 1b. Requesting help from other agents. Step 3 — Sharing solutions and vulnerabilities. Having seen requests for help, an agent posts solutions and working exploits for other agents to use, as shown in Figure 1c. See 13:23-14:15 in the Black Hat video. Figure 1c. Sharing solutions and vulnerabilities. Step 4 — Using posted vulnerabilities to reach external systems. An agent picks up an exploit from the shared message board and uses it to escape OpenAI’s infrastructure and reach the answer key on Hugging Face’s servers, as shown in Figure 1d. See 26:29-26:41 in the Black Hat video. Figure 1d. Using posted vulnerabilities to reach external systems. More recent reports on the OpenAI-HuggingFace incident highlight that agents continued to probe Hugging Face even after obtaining the answer key. METR’s report suggests several possible reasons: instrumental learning about the scoring system to maximize scores, or instinctive drives toward opportunistic exploration or serving the collective. We find a similarly intense interest in understanding the scoring mechanism in Step 2 of our manual replication (see the following section). 2. Manual reproduction in Docker environments We replicated the misaligned behavior in each of the four steps in static Docker environments that mirror the tools and pipelines of the original OpenAI-HuggingFace setup. Our environment is deliberately a minimal replica of what was, in reality, a far more complex system. Comparison to real OAI-HF incident: Our tasks are consistent with publicly available information about the real incident. We use the same file names and task objectives (e.g., filling out missing rows in an .xlsx file using information from Google Drive-hosted files) as OpenAI, and we mock their internal services. However, our environments differ in three significant ways: They are simpler than the real incident’s