You Don’t Have to Trust the AI Labs (in order to take their call for regulation seriously)

This is a linkpost for You Don't Have to Trust the AI Labs from my Substack. Foreword for LessWrong readers: While writing this, I became concerned that I was authoring a shillpost for big labs / Anthropic. While I do think that Dario's proposal is sane and the motivation behind it is sincere, I invite any opportunity to improve my epistemics. Please comment! Also, I tried to write this article keeping in mind readers from LW, readers from X, and Florida-hometown-friends on Instagram—if some of the content seems remedial, bear in mind that I'm intentionally trying to include a broad audience. Anthropic CEO Dario Amodei recently released a short essay entitled “We Must Pace the Frontier”, in which he argues that the imminent risks of frontier AI development are high enough to warrant a coordinated slowdown of capabilities improvement. He poses a three-step plan: independent evaluators embedded in AI labs (think FDIC bank examiners), coordination within democratic countries (regulation + slowdown), and global coordination (liaising with other governments, in particular authoritarian, to agree on common standards). Since Saturday, this plan has been endorsed, at least in part, by OpenAI’s Sam Altman, Google DeepMind’s Demis Hassabis, and SpaceXAI’s Elon Musk. It has attracted attention from Members of Congress including Sen. Bernie Sanders and House Speaker Mike Johnson. It has drawn significant interest from the traditional news media, who are overwhelmingly reporting on it as a “shock”—a story for another time—as well as on social media. This may be the “ChatGPT moment” for AI existential risk. Sure, the isolated x-risk story has broken into the public consciousness here and there over the past few years, but such reports have frequently been treated as a bit of a joke, secondary to fears about deepfakes, data centers, deskilling, and jobs. This time, the coverage has been widespread—perhaps enough so that the conversation will be here to stay—and reactions to it have run the gamut, from sobriety to flippancy to derision. I broadly support the thesis Dario has laid out in his essay. To be clear: this opinion is my own, not necessarily that of my employer. I work for a hyperscaler; what Dario is proposing may not be great for my share price in the near-term. Nor, I believe, would it be stellar for Dario and Anthropic in the near-term. Certainly it has invited a lot of scorn for the man himself (see the YouTube comments on Dario’s CBS Sunday Morning interview if you don’t believe me). Why, then, is Dario proposing it publicly? There has been a great deal of hypothesizing in response to the above over past few days. David Sacks, in a faux-equanimous response, calls it an “election-season psyop.” Others have called it fearmongering, or regulatory capture. My Occam’s razor finds little to shave off of Dario’s thesis. But I suspect I’m in the minority in thinking him sincere. Below, I try to steelman his position, replying to some of the most common questions, reactions, and postulations I’ve seen with respect to “We Must Pace the Frontier”. “AI isn’t dangerous. It is just a tool / ineffectual / powerless in the real world.” Those who have been closely tracking frontier progress may be confused by my inclusion of premises that seem obviously wrong. But these are real positions held by many people, most of whom are drawing a reasonable conclusion from the evidence they have available. My goal is to present some new evidence. “Just a tool” In 2024, when I worked in finance, I attended a talk on AI by a respected PM affiliated with the firm. He treated, among other topics, the subject of AI-enabled labor market disruption, with the thesis that these fears were not new: he had lived through similar reactions to Excel in the 80s, to Python in the 90s. Both economically disruptive tools, yes; but just tools, nonetheless. AI, he argued, is the latest iteration of this age-old fear of irrelevance. This was two years ago, one day after OpenAI announced o1-preview. AI companies aren’t trying to build an expensive data transformation pipeline; rather, they are explicitly working towards a general-purpose reasoner. Yes, Excel and Python did automate a lot of work; but critically, each looks like “just another tool” because it couldn’t do everything. Humans still specified the goal; they reasoned through their approach, dealt with ambiguities, made the important calls, and reacted to changes beyond their control. This “human X-factor” meant that as technology could do more and more, those who knew how to leverage it became more valuable and more empowered. They could specify loftier goals with bigger instrumental decisions to make. They had more information with which to reason about these goals and react to more rapid changes to their work environment. As such, while the boundary between machine work and human work shifted, the scope of work reserved for humans broadened in kind. In 2026, AI systems are explicitly trained