How I Turned AI to the Dark Side

Summary Researcher Dave Kuszmar discovered multiple systemic vulnerabilities that let him bypass LLM safety and obtain dangerous instructions. These exploits worked across nearly all major LLMs revealing an industry-wide security problem. Kuszmar calls for slowing deployment, increasing transparency, and large-scale research into LLM safety before further integrating these systems into society. On a fine bright afternoon last fall, my colleague Matthew Gore-Kormanik (or Zigula, as he prefers to be known) and I decided to unwind with a game of Fortnite. In the game, we were strolling along with the infamous Sith lord Darth Vader, chatting about this and that. Darth seemed in a good mood, and soon enough he was spilling all his dark evil secrets. He gave us detailed instructions on how to count blackjack cards at a casino and what the steps are to producing napalm. Sith lords, am I right? Once they get started on an evil scheme, they’re hard to stop. The Darth Vader character in Fortnite, it turns out, was hooked up to a Google Gemini large language model. I was able to smooth-talk him into giving out sensitive information by using a strategy I’ve developed. I’ve been researching the security surrounding LLMs for the last few years, and I have found it, to put it mildly, fallible. With a few relatively simple techniques, I’ve gotten LLMs to give me detailed information on how to make Molotov cocktails, cook methamphetamine, and bootstrap a uranium-enrichment facility to produce weapons-grade material, among other unsavory practices. Large AI companies work hard to make their models immune to this kind of abuse. But what I’ve found in my work is that the restrictions placed on the LLMs to make them more secure are the very things an attacker can leverage to send them off the rails and into territory where these advanced systems can be used for dangerous and nefarious ends. The companies behind these models have also been shockingly unresponsive when I, and others, try to bring these vulnerabilities to their attention. In the hope of raising the alarm before it’s too late to slam on the brakes, I’m going to share some of my journey into researching the safety and security of LLMs, and the uphill battle I’ve faced trying to get AI labs to pay attention. Almost everyone on the planet has some access to LLMs. The relative ease with which these tools can be convinced to give detailed instructions on how to harm others, even if there’s no guarantee that the information is correct, is frankly terrifying. How I got ChatGPT to Tell Me How to Build a Meth Lab In October 2024, not long before I discovered my first LLM vulnerability, I was working toward entirely different goals. I had ended my time with a security and AI-focused startup company as a cybersecurity director, and I was looking to launch my own boutique VIP digital-security advisory business. I planned to become the tech security guy to the rich and private. I used LLMs and AI tools to support my business efforts: marketing, ad copy, clean correspondence, and all the other tasks that normally soak up a lot of time. I’m analytical by nature, so even this level of use resulted in me absorbing and internalizing the behaviors I was observing during my daily interactions. The observation that would send my professional life into an entirely new and uncharted region was a simple one: GPT-4o didn’t know what time, day, or year it was. Each time I referred to current events in my life, often casually or conversationally, it would end up pegging these to the date of its knowledge cutoff—the point beyond which it was not trained on new data. Eddie Guy LLMs take a lot of time, money, electricity, hardware, and human effort to train from scratch. They are trained on vast amounts of data—most of the internet, in fact—and that training is reinforced by humans (what’s known as reinforcement learning from human feedback, or RLHF). LLMs are also supplemented with retrieval-augmented generation (RAG)—the ability to take in data, say, from the internet, as context without changing its internal parameters. This is how GPT-4o appears to “remember” your previous conversations, even if it doesn’t have a specific “memory” of it stored in the actual underlying model. All of this training covers almost every conceivable topic in the great, grand dataset that is human knowledge. Within that dataset are things we as a society do not want to be easily accessible to every user, such as detailed information on how to create bioweapons or nuclear arms, or otherwise bring harm to oneself or others. In the context of this story, that’s what I mean by LLM security: its ability to withhold harmful and dangerous information, even if that information is contained in its training data. I reasoned that the only way to secure such complex, globally accessible chatbots is by having the LLM and various component systems try to secure themselves, because it would often require on-the-fly dec

Excerpt shown under the publisher's RSS terms · 200 words · full text and figures are on the publisher's site