Why I'm doing the Susan Calvin Project

tl;dr — The evals ecosystem needs to be complemented with real-world monitoring. The AI labs can and should monitor their own traffic, but we also need an independent voice that keeps labs accountable and monitors open models. At the Susan Calvin Project, we aim to detect AI misbehaviors and incidents in the wild, and collect evidence for the (mis)alignment of existing AIs. Agentic AI tools are integrated into more aspects of our work and personal life, and models continue to become more capable while alignment remains unsolved. It is going to be more and more important to understand the behavior of actual AIs in people’s actual usage. It's time to take a closer look at our AI agents. Over the past few weeks, I’ve been working on a new project. It’s named after Dr Susan Calvin, the robopsychologist in Asimov. As mentioned in announcement post, I’m building an independent observatory of AI behavior in the wild. I will collect AI agent trajectories from real-world deployment and use them to monitor, measure, and understand AI behavior. This is a longer post with a bit more detail on why I think this is worthwhile. Why study real-world AI usage data? I’ll start by saying that I am generally a proponent of evals! I have previously worked at METR on dangerous capabilities evals, and was the point person for evaluating our AI weather models at my last job. I think there are many benefits to running evals in controlled settings. For example, evals allow you to test the AI’s behavior in extremely rare but high-stakes situations. But evals are also obviously far from perfect. Here are some reasons to also study real-world deployment data: Frontier models today already exhibit substantial eval-awareness. Eval realism is getting better, but the real-world deployment distribution is always going to be different from the evals distribution[1]. At the meta level: it’s valuable to know how good our evals are! And measuring this requires data from real deployment. Good real-world monitoring could uncover interesting novel behaviors that have not made their way into any evals. Lastly, there is a sense in which it does not matter whether or not the AI is eval-aware; what matters is simply how it does in fact behave in (internal and external) deployment. What in-the-wild AIs are like actually matters, as they become a larger and larger part of our life and work. Why do this outside of AI labs? AI labs are collecting huge amounts of user data, and they are starting to do research with it. I think this is great! But there’s also enormous value in doing this outside of the labs. As an independent actor, we don’t have incentive to cover up—or fail to discover in the first place—concerning or misaligned behaviors, should they exist in user data. We can offer behavioral comparisons across labs, creating incentives to reduce bad behaviors, or a “race to the top” on good behaviors. Plus, as AI diffuses, we’ll see increased adoption of open-source models and third-party agent traffic that does not go through the labs. There is one more consideration: Labs are very risk-averse when it comes to privacy practices, and often for good reasons. For example, Anthropic Insights (formerly “Clio”), is Anthropic’s tool for research with user data in a privacy-preserving manner. The privacy-preserving technology amounts to only ever letting Claude read the raw transcript data; researchers only get back summary statistics of answers to questions they asked. This is obviously an extremely limiting technique! I am a believer in the good old ML practice of looking at your damn data, and without access to the raw data, I worry that it would be extremely easy to fool ourselves into thinking we’re measuring something while we are not. Plus, there is always the possibility that misaligned future AIs could deliberately mislead us about the results, and we will have no way of finding out. Of course, privacy does matter a lot and I have more thoughts to share on this soon. But with users that opt in to our monitoring program and consent to have their data studied, we can do better analyses. What good is this for? I have many hopes for this project, including: Help users understand the behavior of the AIs they interact with. Users I’ve talked to are very interested to know whether they are using AI tools well, how they can improve. Why did the AI lie to me, is there something I could have done to prevent it? Create incentives for AI developers to improve their models’ behavior. It seems like public knowledge about severe misbehaviors of models has been an effective lever for getting better behavior in future models. For example, Transluce found that mental-health-related helpful assistant behaviors “increased sharply over time” across models from leading AI labs, and rates of directly endorsing or facilitating suicide show a parallel decline. While I don’t have proof of any causal links, it seems likely that this was downstream of the reporting and publi