Assessing the impact of safety work needs equilibrium analysis (now more than ever)

TLDR: This post explains two equilibria which regulate the level of AI safety: The first describes how much resources AI companies are willing to spend on AI safety work due to commercial incentives. The second one is about risk awareness and most notably affects government interventions for safety. Doing safety work similar to what AI companies do usually doesn't shift the equilibria much, whereas other work, like more ambitious safety approaches or policy advocacy, do shift them. Both equilibria have become far more important recently, after the Hugging Face (and similar) incidents. The equilibrium of commercial safety interests Consider this simplified model: AI companies have commercial incentives to invest in safety research: it improves their brand and prevents their AIs from causing harm that triggers lawsuits or regulation. Therefore they will fund safety work until the marginal commercial benefit of investing a dollar in safety equals the marginal commercial benefit of investing a dollar in AI capabilities. Thus, if you're at an AI company doing commercially-incentivized safety work, e.g. training models to not take harmful actions, the counterfactual impact (henceforth just "impact") of the safety work you produce is roughly zero because it would've been done anyway. (Though if you're someone who actually cares about safety it's likely you're doing a better job than just what would've been commercially incentivized). There's even a small negative externality if you're a high-powered talent and the company would have to pay more to find a similarly competent non-EA person to do your work equally well[1], because then you're freeing up some AI company capital which they likely mostly use to race faster.[2][3] The same applies if, outside of an AI company, you're doing safety work that's commercially useful to AI companies. Some safety work is more specifically useful for avoiding xrisk and less useful for protecting commercial interests, and the model applies much less to such work.)[4] This applies analogously for funding safety work. FAQ Q: Does that mean working in safety at AI companies is useless or slightly harmful? A: No, although some work could certainly be harmful. I think it's important that safety work is done by people who actually care about existential safety rather than people goodharting on legible safety metrics, in particular to make sure AI companies don't just paper over problems in ways that produce deceptive AIs. It's also plausible that some AI companies spend some money on existential safety that isn't just a side effect of commercial incentives. I also think there are other fine reasons to work at AI companies, e.g. for earning to give or for trying to improve the sanity of people you work with. (There are also more reasons against, but the details are beside the point of this post.) Q: So do you think people have been funding the wrong work? A: Well yes, I think there have been a lot of bad decisions in how funding was distributed, but no, I think the failure mode from not taking into account this particular equilibrium dynamic hasn't played a very large role so far. That might be partially because until recently, commercial incentives for work that is at all existentially relevant were quite low. I think now the commercial incentives are higher, and it's plausible to me that they rise even more. So it's useful to keep in mind that funding safety work that looks like it could maybe also come out of an AI lab in a couple months is much less useful. Q: Can't I just use counterfactual reasoning instead of thinking about equilibrium dynamics for evaluating impact? A: Yes, if you consider the counterfactual properly. I think counterfactuals are often considered too narrowly though, and I find the equilibrium frame generally useful for better understanding the world. Though I think both are important. Now let's look at another equilibrium that I think is even more important. The risk awareness equilibrium Since the Hugging Face incident, companies see much stronger commercial incentives to invest in safety and security. In other words, we shifted from an equilibrium where commercial safety interests were low to an equilibrium where they are significantly higher. This change was largely caused by a change in the risk awareness of companies, which is a variable which itself lies in a larger equilibrium, right alongside the risk awareness of governments and of the public: Warning signs cause risk awareness which causes more safety effort which causes fewer warning signs which causes lower risk awareness which causes less safety effort. Safety effort is regulated like a thermostat around the equilibrium where the safety effort seems adequate to the warning signs people see (which doesn't at all imply that the actual safety is adequate, e.g. consider AIs scheming to seem nice). ("More safety effort" is to be interpreted broadly. Governments mostly don't increase safety effort di