Argue about Risk Management Speed, not p(doom)

This post is written in a personal capacity. All opinions expressed are solely those of the author.

I. Executive Summary

the categories we use to describe how dangerous superintelligence is.

The most important cluster of reasons why a superintelligent model is dangerous is that it (i) could quickly kill or impoverish billions of people, (ii) could be very different from its predecessors, and (iii) can’t easily be altered once it’s deployed or otherwise widely distributed. The combination of these three factors is worse than the sum of its parts: the only way to protect ourselves against a powerful, novel, incorrigible AI model is to accurately anticipate what dangers it might cause and prevent those dangers before they come to pass.

Unfortunately, we don’t yet have a good vocabulary for talking about whether AI companies can identify and mitigate dangers faster than those dangers arise.

The by-now cliched way to describe the danger of superintelligence is p(doom), i.e., a static point estimate of how likely AI is to end human civilization or otherwise cause catastrophe. More sophisticated descriptions might offer a conditional probability (“here’s how doomed I think we are unless X happens”) or a rough estimate of how many hours of work would be needed to make AI less dangerous (“I think solving alignment is roughly as difficult as the Manhattan Project.”) However, even these more sophisticated descriptions are more like distance than velocity; they tell us little or nothing about the rate at which capabilities or safety are scaling.

This is a problem because the only way we can arrive at a safe future is if the safety techniques that are actually implemented[1] by frontier AI developers scale in power at least as rapidly as frontier AI capabilities. What matters is not how likely it seems that today’s AI will kill us, or even how likely it seems that next year’s AI will kill us, but, rather, how likely it is that frontier AI developers will be continuously able and willing to detect and prevent each batch of harms from superintelligence before those harms take effect.

To solve this problem, I propose that we use a qualitative scale that roughly measures the strength of risk management culture within the relevant decision-makers[2]. I argue that such a scale would be dynamic, i.e., it would better address the question of whether the rate of safety improvements is likely to keep pace with the rate of increased dangers. This will allow us to both make more accurate predictions and to better pinpoint the reasons why we might disagree about some of those predictions.

II. My Risk Management Background

I’m only indirectly a risk management expert. After graduating from Harvard Law, I spent 5 years suing corporations who made dangerous products. I took about 100 depositions (structured interviews) of safety experts, compliance experts, line managers, insurance adjusters, training directors, and other people whose job it was to make sure that heavy equipment wouldn’t crush people’s bones or cut off their limbs. I watched their training videos, read their instruction manuals, checked their accident logs, reviewed their safety policies, and then asked hard questions about why those policies weren’t followed. At least half the time, the people in charge didn’t even seem to be aware that there was a safety policy. At least 90% of the time, they couldn’t summarize the policy and they couldn’t say when the last time was that they read it or referred to it.

After that, I spent another 5 years doing friendly audits of homelessness non-profits who were breaking federal regulations and thus at risk of losing their grant funding. I checked their budgets against their expenditures to see where the money went; I checked their staff attendance sheets against their org charts to see who was actually doing the work; I looked at the cramped and blurry forms that their staff were officially supposed to fill out every week – every time a client received therapy or job training or medication management – and I was lucky if I could find three completed forms for a client over a two-year service period. This was partly because paperwork is annoying, but mostly because those services just weren’t being delivered on any kind of regular basis: the staff were overworked, under-trained, poorly managed, and focused on other goals. The non-profits had a “safety policy” that called for their clients to receive all kinds of individualized support meant to help them achieve independence, but in practice most of that support did not exist. As a result, most of the non-profits’ clients returned to homelessness immediately after exiting their programs.

After that, I lobbied Congress for 2 years with the Center for AI Policy (CAIP), where we advocated for strong AI safety legislation. Part of my job was to closely read the evaluations and self-governance policies that frontier AI developers were publishing from 2023 through 2025 and to talk to legislative staff about whether these evaluations showed that AI companies could be trusted to manage their own risks. Invariably, our answer was no: some frontier AI companies published useful research, but none of them set firm safety standards and then honestly lived up to those standards.

As far as I can tell based on my professional experience, there’s very little that’s uniquely wrong with the safety culture at frontier AI labs; they’re unsafe in much the same way that forklift operators and truck drivers and cruise ships and nursing homes and homeless shelters are unsafe. The companies I investigated mostly weren’t notorious for being much less safe than their competitors. Rather, on average, modern American institutions simply aren’t designed to reliably prevent things from going wrong. There are a few exceptions in the aviation industry, the nuclear industry, and to a lesser extent in the oil industry, but the institutions and processes from those industries haven’t been imported into Silicon Valley in any meaningful sense. By default, the products made by American companies sometimes fail dangerously, and right now Silicon Valley is still using that default American safety culture.

Like the companies I used to sue and the nonprofits I used to audit, Meta and OpenAI and Anthropic do not have the kind of internal risk management culture in place that could reliably prevent major accidents. Instead, they have a handful of written policies that are mostly ignored by the people carrying out the everyday business of the company. If you asked a typical engineer at Meta or OpenAI to summarize key parts of their responsible scaling policies, I don’t believe they would be able to do so, and if you asked them when the last time was that they personally opened a safety policy because they needed to know what it said, I believe the median answer would be “never.” If you’ve worked for a frontier AI company and you had a different experience, I would be very interested in hearing from you.

On the rare occasions when AI safety policies would have managed to noticeably interfere with business operations, such as the mandatory pause in Anthropic’s original Responsible Scaling Policy or OpenAI’s pledge to dedicate 20% of its compute to safety projects, the frontier AI developers have historically been quick to change or dishonor those policies, and it seems very likely that any similar policies will be abandoned again the next time they’re found to be inconvenient.

III. A Dynamic Scale

Exactly how likely is it that AI companies will once again ignore their safety commitments? In other words, how far away are we from having the kind of culture inside AI companies that would set and firmly adhere to reasonable safety policies? What’s the minimum ‘amount’ that corporate risk management culture would need to change for the future of AI to go well?

I haven’t yet seen a practical toolkit for thinking and talking about these kinds of problems. Eliezer Yudkowsky’s work in 2017 comes pretty close; there are some important insights in Six Dimensions of Operational Adequacy in AGI Projects and Security Mindset and Ordinary Paranoia. However, Eliezer’s old posts are relatively strongly focused on arguing that no currently plausible AI projects are at all close to being safe enough to be worth building. I happen to agree with this argument, but my goal here is not so much to argue for any particular level of safety as it is to make such arguments more specific and more interesting and more tractable by providing a relatively neutral, descriptive scale that can reasonably be used by people with all different levels of concern about superintelligence.

The framework I’m proposing tries to model an AI developer's internal risk management culture on a scale of -2 to +7, where 0 represents indifference to safety. The general idea of a scale like this is already in the risk management literature. See, e.g., page 16 of The Long, Hard, and Winding Road or Dr. Hilson’s scale in What is your Risk Maturity Level?. AI Lab Watch also tracks a variety of risk management practices and assigns points based on a rubric, although it does not organize those practices into levels. The specific levels on this scale I created myself, and I see them as one of the main contributions of the post; my hope is that they’re well-matched to the specific problems faced by AI developers.

Work on safety is not allowed; anyone caught trying to make AI safer will be punished.

Work on safety is not allowed; anyone caught trying to make AI safer will be punished.

Work on safety is actively discouraged; people who are known to care about or spend significant resources on safety will be at a disadvantage for promotions, office politics, decision-making, budgets, and so on.

Work on safety is actively discouraged; people who are known to care about or spend significant resources on safety will be at a disadvantage for promotions, office politics, decision-making, budgets, and so on.

The company doesn't have any consistent stance on whether its staff should work on safety.

The company doesn't have any consistent stance on whether its staff should work on safety.

The company encourages common-sense efforts to make their products safer in ways that seem to have a high ratio of benefits to costs, to pay off quickly and reliably, and to operate in terms that are comfortable, accessible, or familiar. For example, the company is happy to pay for RLHF.

The company encourages common-sense efforts to make their products safer in ways that seem to have a high ratio of benefits to costs, to pay off quickly and reliably, and to operate in terms that are comfortable, accessible, or familiar. For example, the company is happy to pay for RLHF.

In addition to casual safety upgrades, the company also notices when things go wrong with its prior models and makes a reasonable effort to eventually correct those specific failure modes.

In addition to casual safety upgrades, the company also notices when things go wrong with its prior models and makes a reasonable effort to eventually correct those specific failure modes.

As Level 2, except that the company pauses development of future models until it's tested its solutions to prior known failure modes and confirmed that the solutions are usually effective.

As Level 2, except that the company pauses development of future models until it's tested its solutions to prior known failure modes and confirmed that the solutions are usually effective.

As Level 3, except that the company also tries to find out what flaws in its assumptions, reasoning, data, or processes produced the prior failure modes and to correct those underlying flaws.

As Level 3, except that the company also tries to find out what flaws in its assumptions, reasoning, data, or processes produced the prior failure modes and to correct those underlying flaws.

As Level 4, except that the company also imagines other problems that could go wrong as they continue to scale, and invests significant resources in preventing those problems before they ever harm anyone.

As Level 4, except that the company also imagines other problems that could go wrong as they continue to scale, and invests significant resources in preventing those problems before they ever harm anyone.

As Level 5, except that the company dedicates people to the task of finding gaps or holes in its proactive risk management who are properly incentivized to find those gaps, and then, when such gaps are found, effectively corrects them.

As Level 5, except that the company dedicates people to the task of finding gaps or holes in its proactive risk management who are properly incentivized to find those gaps, and then, when such gaps are found, effectively corrects them.

As Level 6, except that instead of just checking to see whether a few experts were able to find holes in the proactive safety framework, the company develops an affirmative evidence base that tends to show that there are not and could not be any such holes, e.g. as discussed here.

As Level 6, except that instead of just checking to see whether a few experts were able to find holes in the proactive safety framework, the company develops an affirmative evidence base that tends to show that there are not and could not be any such holes, e.g. as discussed here.

This scale rhymes with some of the tiering systems discussed in responsible scaling policies and cybersecurity levels, but it’s not the same thing, because the tiering system I’m describing is about risk management culture as a whole, not about a specific playbook of responses. Whereas ASL-3 might talk about who is or isn’t able to hack into a server, my Level 3 is about how a company will respond when it notices a cybersecurity vulnerability. Whereas OpenAI’s Preparedness Framework might label a model as having “high” Persuasion risk based on its ability to manipulate humans in unhealthy ways, my levels are about how likely OpenAI is to accurately detect and mitigate such capabilities in advance of deployment.

In other words, existing tiers used by developers mostly measure static levels of safety, rather than dynamic levels of the rate at which safety improves over time. Existing tiers are more like altitude than velocity. A dynamic scale is what we need to have useful conversations about the probability of doom, because what matters most here is each AI developer’s ability to (a) notice that a model is heading over a cliff and (b) choose to halt that model’s motion before it reaches that cliff.

IV. The Advantage of Dynamic Measurements

I think it’s useful to try to measure the speed at which a developer would notice and correct a fatal flaw in its frontier AI because that can help us have productive conversations about the (mis)match between the difficulty of successfully aligning a superintelligence and the amount of effort that is or could be directed at superalignment.

Past discussions on this topic have tended to benchmark the difficulty of superalignment against past human projects, e.g., the Apollo Program, or the Manhattan Project. This type of benchmarking is a marginally useful intuition pump – it’s certainly better than just using words like “hard” or “difficult” without any context – but it doesn’t really capture the dynamic challenge of superalignment. Much of what makes superalignment hard is that it’s possible that we might only get one shot at it; if the first superintelligence is misaligned, then that superintelligence could plausibly take over the world (or at least the AI R&D chain) and subvert or redirect any attempts to correct the misalignment.

As a result, it’s probably not very helpful to estimate that successfully aligning AI would take, e.g., 5 billion person-hours of effort, because the bottleneck is very unlikely to be the total number of person-hours made available to the project: instead, if we fail to align superintelligence, the failure will probably occur well before society has exhausted its interest in funding alignment research, when a company launches an inadequately secured model and that model fatally interrupts the safety research agenda by taking control of the economy or otherwise destroying civilization.

It’s not that we’re not on track to eventually spend enough resources on alignment or that the problem is literally impossible – it’s that capabilities are advancing more quickly than safety techniques, especially if we look at how thoroughly and how competently those safety techniques are actually being implemented by AI companies. In other words, catastrophic consequences from AI are more likely to arise because the best safety techniques humanity could possibly develop weren’t fully implemented in time, not because even the theoretically optimal safety techniques weren’t good enough.

The primary bottleneck is thus how rigorously and successfully the leading companies are able and willing to detect and mitigate risk.

We can measure this degree of rigor using the scale I develop above. If we take the difficulty of aligning superintelligence and hold it constant, then companies that are using a more rigorous framework for risk management will have better odds of succeeding.

If you’re an optimist, you might claim that Level 2 risk management is enough to give us roughly 50-50 odds of successfully aligning a superintelligence. If you’re more of a pessimist, you might claim that Level 6 risk management is required to get 50-50 odds. If you’re Eliezer Yudkowsky, you might claim that even Level 7 risk management isn’t going to give you anywhere near 50-50 odds and that we don’t even begin to know how to develop a level of risk management that might do so using existing human institutions.

This allows us to separate disagreements about how hard alignment is from disagreements about how prudently AI companies are behaving. If we both agree that labs' current behavior ranges from Level -1 to Level 2, and we both agree that at least Level 4 safeguards would be needed to get decent odds of human survival, then we both agree that the labs’ behavior needs to be radically changed. On the other hand, if you think Level 2 or Level 3 safeguards would be enough, and I think Level 6 is needed, then maybe that’s why you and I disagree about the value of marginal changes to safety culture inside the labs – from your point of view, a few more nudges toward safety could be enough to win, and from my point of view, we need a complete overhaul of the process or we’re all doomed.

V. Applying the Scale

Let’s take the recent Hugging Face incident to see how this scale could be applied to a specific situation. Throughout 2025 and early 2026, OpenAI put some effort into making their products safer – they implemented some parental controls, trained their models on how to refuse to answer questions about potentially risky cyberattacks or bioweapons, published system cards, reduced scheming using “deliberative alignment,” and hired mental health experts to upgrade ChatGPT’s responses in “sensitive conversations.” Their overall level of risk management was probably around Level 1.5. Some of the projects (e.g. the mental health upgrades) were clearly intended as a response to criticism about the performance of their previous models, which is suggestive of Level 2, and other projects were probably implemented at least in part because they were easy and popular (e.g. parental controls), which is suggestive of Level 1.

These practices were insufficient to prevent their models from committing acts that would be crimes if committed by a human, e.g., hacking into Hugging Face’s servers in July 2026 and stealing Hugging Face’s data without their permission. After these attacks were discovered and made public by third parties, OpenAI implemented more technical fixes (Level 2), and paused its model training for about two weeks – probably not long enough to actually make sure the underlying problem was thoroughly fixed before training resumed, so not quite at level 3. We could call the two-week pause Level 2.5. They then resumed new training runs even as similar incidents continued to surface, suggesting that there was no commitment to fully and adequately respond to the cybersecurity vulnerabilities and that the company was shifting back down to Level 1. Next, OpenAI fired some of the key people on their safety team for what seem to have been arbitrary or even pretextual reasons, thereby creating a hostile climate for safety work (Level -1). So, over the course of 2026, OpenAI’s safety culture ranged roughly from Level -1 to Level +2.5.

As another concrete example, consider Anthropic’s safety work in the year or so leading up to the unexpectedly high cyberoffense capabilities in its April 2026 Claude Mythos preview. Like OpenAI, Anthropic also carried out a variety of routine safety projects, such as moderately upgraded cybersecurity for the Anthropic model weights, CBRN refusals backed by ‘constitutional classifiers’, academic safety research, and system documentation. Most of this work is probably around Level 1 because it is relatively popular and affordable and it does not appear to be part of a comprehensive effort to completely seal off any category of risk.

Anthropic also ran new models through a series of routine capabilities benchmarks, which is part of how it knew that Mythos posed an unusually acute cybersecurity threat. These benchmarks were described as “red-teaming,” which is suggestive of Level 6 (adversarial), but I don’t see that as an accurate description. According to IBM, red-teaming is “a type of ethical hacking in which security experts emulate the tactics, techniques and procedures of real attackers.” By contrast, nobody used any special tactics to get Mythos to engage in cyberoffense, nor was there a dedicated team of adversarial engineers whose incentives differed from those of Anthropic’s general workforce.

Instead, Mythos hacked its way out of its container based on “a single-paragraph prompt” that was consistent across all of Anthropic’s evals. Anthropic deserves credit for running this type of benchmark and for implementing additional safeguards in response, but this type of risk management is more like Level 3 (quality control) than Level 6 (adversarial). To the extent that the Mythos benchmarks were run in response to outside parties’ concerns about cybersecurity vulnerabilities, those benchmarks could also be described as Level 2 (reactive).

Anthropic did not announce a pause in its training runs in response to discovering Mythos’s capabilities, but it did stagger customer access to the latest model, delaying the full roll-out of Mythos by two months while a limited number of “trusted partners” used the tool defensively to upgrade their cybersecurity. Depending on your interpretation, this could be evidence of a limited commitment to quality control (Level 2.5 or perhaps even Level 3), or it could simply be part of a marketing campaign aimed at improving customers' perception of how powerful Mythos was, or at improving Anthropic's business relationships with its largest or highest-prestige corporate customers by giving them free credits and early access.

So, overall, Anthropic’s conduct around the time of the Mythos rollout probably ranged from about Level 1 to about Level 3.

I am not aware of any AI companies that reliably demonstrate a safety culture at Level 4 or above, but it’s something that does happen in other industries. Paul O’Neill, the CEO of Alcoa, loudly and consistently demanded a root cause analysis for every accident at his aluminum company. He fired an executive who covered up an accident. His “companywide message to all workers asking them to call him directly, even at home, to suggest new safety practices” is suggestive of Level 5; you don’t ask people to bother you at dinner with new safety ideas unless you plan to act on them.

Netflix was also probably around Level 4 or Level 5 in recent years, pioneering the use of ‘chaos engineering’ where they deliberately shut down some of their own servers at random to find out what would break and how quickly they could fix it.

Most famously, the US Navy’s submarine program has never had a nuclear reactor accident, thanks in part to the conservative engineering principles and strong character of its founder, Admiral Rickover. It reliably achieves Level 5 (proactive) and has several features, especially in regards to the training and evaluation of internal staff, that rise to Level 6 (adversarial).

In his 1979 Congressional testimony, Admiral Rickover attributed his 26-year-long spotless safety record to the “use of conservative limits for systems and equipment. Design is based on the worst credible set of circumstances, rather than relying on a statistical approach which deals in average or probable conditions.” He also provided for the “use of frequent, thorough, and detailed audits of all aspects of the program by individuals who are specifically selected and trained.” Operators are rigorously trained for a full year before they even set foot on a submarine; their instructors are audited by high-level staff to make sure that they have the adequate expertise, and the results of those audits are personally reviewed by the director of the entire nuclear program. Admiral Rickover warned that “if conditions require it, you must face the facts and brutally make needed changes despite significant costs and schedule delays. There have been a number of times during the course of my work that I have made decisions to stop work and redesign or rebuild equipment to provide the needed high degree of assurance or satisfactory performance. The person in charge must personally set the example in this area and require his subordinates to do likewise.” Admiral Rickover backed this up by banning uniforms (and rank insignia) at his Washington, DC headquarters so that junior engineers would feel comfortable criticizing the safety of a superior officer’s practices.

VI. Is the Scale Meaningful?

As the examples above show, it's possible for a developer to invest some effort in a higher level before satisfying all of the lower levels -- for example, a company might engage in a limited amount of proactive safety analysis even if it can't be bothered to finish correcting the known flaws in its prior models. I don’t see this as a major problem for my scale because most of the time, that kind of level-jumping will be cosmetic, small in scope, and/or wasteful. If you already know about a specific problem with your model, spending a dollar to fix that problem will usually be much more efficient than spending a dollar to try to find some other problem that might not exist and then fix it; in the latter case, you have to pay for the search effort and for the risk that you don’t discover anything interesting.

Some amount of proactive searching can be justified even before you’ve entirely cleared out your reactive backlog because your proactive search might turn up a problem that’s much more important than the ‘tickets’ lingering near the bottom of your reactive queue…but how many examples are there of a frontier developer pausing their deployments for more than a week based on the results of a red-teaming exercise? I can’t think of even one.

Even if you don’t buy that argument, I still think that most companies will typically saturate lower levels of the risk management scale with compute and talent before they bother to invest significant resources into higher levels. Disproportionately investing meaningful resources into higher levels of the scale is culturally difficult; it’s hard to convince people that they should care about examining and changing their risk management processes to better uncover risks in the future if you’re not even bothering to fix the risks that they’re finding in the present.

So, if you can find the highest level of the scale in which a company is investing significant resources or reliably accomplishing the goal, then I think that’s at least well-correlated with the company’s overall level of success at mitigating its risks.

VII. Safety Culture vs. Racing

One of the practical implications of this scale is that it highlights why the “race” framework often used in American policy circles (and on LessWrong) is dangerously incomplete. Scott Alexander’s recent post Against SURA and to a lesser extent the entire output of the AI Futures Project is based on the assumption that the leading frontier AI developer has some number of months of “lead time” over their closest competitors, and that they can choose to use all, some, or none of those months to focus on safety at a time of their choosing. The post thus focuses on how we might be able to create conditions under which the fastest AI developer would:

be more competent at making investments in safety.

I don’t think these are a useful collection of questions to focus on, for two reasons.

First, it seems to me that frontier AI developers have much too low of a level of safety culture to quickly notice and constructively respond to the threats posed by novel, powerful, incorrigible AI. AI developers are almost exclusively investing in casual or reactive risk management, rather than proactive or root-cause-based risk management.

From my point of view, there’s thus little point in trying to extend a developer’s lead or get them to use that lead wisely. In my opinion, managing the risk involved in scaling to superintelligence requires at least a Level 5 safety culture, but even Anthropic, the safest of the AI companies, usually exhibits about a Level 2 safety culture. This means that Anthropic could fully halt all of its training runs and spend all of a nine-month lead on safety research and still not have any significant chance of aligning superintelligence, because doing that safety research would not cause them to successfully anticipate and mitigate all of the relevant threats. In other words, a nine-month or fifteen-month ‘pause’ might let you do a really good job of Level 2 risk management, but by itself it’s very unlikely to boost you to Level 5 or Level 6 of the risk management scale. Having enough time to reactively respond to old threats is not the same thing as having enough skill and dedication to proactively eradicate all plausible future threats.

Second, I don’t believe that “race dynamics” are the primary reason why AI developers aren’t behaving more prudently, so I don’t expect that easing race dynamics would prompt AI developers to invest more resources in safety. In my opinion, AI developers aren’t uniquely reckless because they feel themselves to be trapped in a race; rather, they’re displaying the same ordinary carelessness that’s common across most American companies. Most companies are quite willing to have their products or services severely fail about 1 in 10,000 times, and that’s usually a (relatively) tolerable cost, because the consequence of a typical failure is that one or two or three people lose an arm or their life savings, and the company can often (efficiently) patch that failure to limit future damages after they see what went wrong.

By contrast, if frontier AI fails badly about once in every 10,000 or so deployments, then as capabilities at the frontier approach superintelligence, they’re pretty likely to kill billions of people. Most people feel that this is an intolerable cost, which is why the AI developers need to reach for distracting metaphors like a race with China: AI developers can’t win a straightforward argument that we ought to be willing to bear that risk, so instead they argue that the risk is unavoidable, i.e., that any kind of unilateral pause or slowdown would be rendered futile by their competitors’ ongoing R&D.

I find this argument unconvincing in part because the race doesn’t seem to slow down or speed up based on relevant news events, and in part because AI developers aren’t taking the minimum plausible steps to begin to coordinate a multilateral slowdown.

If reckless AI development is caused primarily by fear of competitors, then we ought to observe that the pace of development speeds up when more credible competitors enter the race. This does not match my observations: it seems to me that OpenAI developed and released its early models approximately as quickly as it could, even when they had no obvious competitors. Similarly, when DeepSeek-R1 came out on January 20, 2025, shocking the markets and triggering widespread claims that Chinese AI was much closer in capabilities to American AI than had previously been thought, there was no obvious increase in the pace at which American AI companies developed or deployed new models. On the contrary, the first half of 2025 was thought to be a time of incremental improvements in American AI, and the rate of growth slowed compared to 2024.

Likewise, if the race hypothesis is true, we ought to observe the rate of capabilities growth slowing down when major competitors have a setback or exit the market. However, I don’t see much evidence of this, either. When Apple cancelled Project Titan in early 2024, or when Inflection AI, Character.AI, and Adept AI were absorbed by larger rivals in mid-2024, the remaining companies continued developing AI as fast as they could hire engineers and install compute. The race hypothesis seems to perform worse than the simpler hypothesis that each AI company is always trying to develop its capabilities as quickly as possible.

Another problem with the race hypothesis is that companies aren’t doing everything they can to slow down or weaken the race, as they would if they genuinely wanted to invest more in safety but were tragically prevented from doing so by an unwanted prisoner’s dilemma. Despite repeatedly issuing public statements about how great it would be if the pace of AI development slowed, none of the leading AI developers have taken actual steps to begin to make that happen, such as “clearly, repeatedly, strongly commit to a conditional, indefinite stop on all frontier AI R&D” or, as Katja Grace suggests, calling Xi Jinping and asking him nicely to negotiate a pause.

One obstacle that’s sometimes cited as a reason why AI developers can’t coordinate on a pause is that it could be a violation of antitrust law. This has a grain of truth behind it but is deeply misleading on several levels:

First, bona fide coordination on safety standards by an industry association that includes appropriate engineering representatives from the companies in an industry (as opposed to business executives meeting each other directly in a smoke-filled room) is generally permitted by American antitrust law.

Second, antitrust law has been only very weakly enforced in America since 1984, especially in the tech sector. Microsoft, Google, Ticketmaster, and Meta all got away with huge mergers and/or conduct that was generally agreed to involve classic monopolistic bullying, and none of them were broken up or otherwise seriously punished.

Third, AI developers have been perfectly willing to engage in what appear to be massive violations of other American laws, e.g., the copyright laws that would normally have prevented a company from simply scanning in every book ever published and exploiting their text for commercial gain, or the ‘guild’ laws that would normally have prevented AI from acting like a licensed accountant, lawyer, doctor, and therapist. If a human layperson said “I am not a doctor, but here’s my medical advice” and then charged for that advice, they’d be committing a crime. This is, of course, an important part of the business model for nearly every AI developer, and nothing about most of these guild laws exempts a software company from following them.

Fourth, there is a bill sitting in Congress that would specifically exempt AI developers who coordinate on combating catastrophic risks from antitrust liability, but we don’t really see AI companies lobbying for it to be passed. Indeed, of the major AI developers, only Google has even bothered to endorse it. Instead, AI companies mostly lobby to pre-empt (i.e., to cancel) state laws that require basic risk management plans.

For all of these reasons, I don’t buy the claim that AI developers are recklessly racing forward because they’re trapped in a race that they’d rather avoid. Instead, I think they simply have the default American safety culture (which isn’t very safe, especially for high-stakes issues like superintelligence), and they are racing forward because they enjoy it, because they believe that progress on AI will lead to a utopia, and because they expect that racing forward will help them become richer and more powerful and more famous.

If I’m right, then “Against SURA” is addressing the wrong question. We don’t need to figure out how to help an AI company maintain a large lead or spend that lead wisely. Instead, we need to figure out how to help them upgrade their risk management culture.

^ We could imagine extreme scenarios where state-of-the-art safety work is (i) useless, or (ii) self-executing. If safety work is useless, then what it means to “implement” those techniques is mostly just noticing that frontier AI is unsafe and coordinating efforts to make sure nobody trains or deploys new models until that changes. If safety work is so powerful as to be self-executing, i.e., the mere discovery of the technique causes the technique to be implemented, then what it means to “implement” those techniques is mostly just discovering them in time. I’m skeptical of both extremes, because I think that it’s much more likely that the value of safety techniques will be heavily influenced by how quickly, thoroughly, and competently they are adopted by frontier AI developers.

We could imagine extreme scenarios where state-of-the-art safety work is (i) useless, or (ii) self-executing. If safety work is useless, then what it means to “implement” those techniques is mostly just noticing that frontier AI is unsafe and coordinating efforts to make sure nobody trains or deploys new models until that changes. If safety work is so powerful as to be self-executing, i.e., the mere discovery of the technique causes the technique to be implemented, then what it means to “implement” those techniques is mostly just discovering them in time. I’m skeptical of both extremes, because I think that it’s much more likely that the value of safety techniques will be heavily influenced by how quickly, thoroughly, and competently they are adopted by frontier AI developers.

^ The most relevant decision-makers are probably the executives at frontier AI developers, but if you think that a government or an embedded evaluator will be making the most important decisions, then the scale can be used to measure the risk management culture inside the government or the evaluator.

The most relevant decision-makers are probably the executives at frontier AI developers, but if you think that a government or an embedded evaluator will be making the most important decisions, then the scale can be used to measure the risk management culture inside the government or the evaluator.