Bottlenecks in AI Safety - Opportunities for Impact

If you are competent, driven, and can get stuff done, you can help unblock this. Cross-posted from my Substack. Written primarily for people in competitive industries who are concerned about AI safety and wondering whether they are suited for having impact, and if so how they can go about entering the field. Any critiques of the substance or arguments here are welcomed. TL;DR Why this post I spent the last 2 years as a quant trader. This summer I did MATS (an AI safety research programme) while on sabbatical. I spent most of the programme expecting to go back to trading. Then the Hugging Face incident happened, and I updated quite hard on a lot of things and decided to work on reducing catastrophic AI risks. This post is written for competent, driven people working in competitive industries who are concerned about these risks - to argue that they are set up for having high impact given the state of the field right now, and to provide an outline of how to actually go about doing this. Primary arguments One objection I've heard from various people lately: "Catastrophic AI risk in the next few years is maybe 10–30% (I updated a lot after the Hugging Face Incident). But I have no experience, and large N (thousands) of people already work on this, so my expected impact is roughly 1/N. I'm comfortable and well paid. Not worth it." The 1/N approximation is wrong. There is no conservation law making individual counterfactuals sum to the group total. Your counterfactual impact depends on the structure of the field, and right now: The field is not constrained by money, talent, or ideas. Indeed, substantially more funding is about to come online as Anthropic IPOs and DAF money unlocks. The field is short of competent, agentic people who can either onboard and work fairly independently fast, or absorb and utilise those resources: founders, team leads, people who can get good work done with high independence, and can build. Eg MATS is scaling ~2x per year and is bottlenecked on organisational capacity - not funding, not mentors, not technically talented applicants. My rough estimate is: ~400 people per year attempt to build new safety orgs, against 1,000–5,000 working on AI risk in total. The former is fairly neglected, meanwhile the implied demand (resource overhang and explicit push for more founding) is high. If you've done good work in a competitive industry - run things, built things, worked under pressure and with high independence - you are differentially suited to have impact right now. Exploration is cheap: ~30 hours to get properly informed about risks, maybe 60hrs to do substantial further derisking such as a small project, 6–12 months to test working in the field more seriously, and something similar to your old job (or its equivalent) is still there if it doesn't work. The downside of waiting is that by the time the risk is legible enough to convince you, it may be too late to act on it. How to read this post Broadly convinced already and wanting to know how to get started → skip to Problems to Target and What Next. Unsure of impact or state of the safety field right now → 1/#ppl is wrong, State of the Field, and Supporting Arguments. New to AI risk arguments entirely → start with the info sources at the end, then come back. If you want to talk, you can ask a question with your name and email here. Why am I writing this As mentioned above, I recently left my job at a trading firm to work on AI risks, having been aware of them in the background for a while. I was skeptical of theoretical risk arguments actually realizing (which I had thought required actors to take ~bad decisions), and quite reliant on the world acting appropriately in response to risks as a means of avoiding bad outcomes. For the majority of MATS, I was unconvinced that I would work on safety. Then, the OpenAI Hugging Face hack and surrounding details happened. This updated me quite a lot on: The decision-making of frontier AI labs and the output of the race & winner-takes-all incentives The state of and value of strong non-lab safety researchers; Ryan Greenblatt was one of three investigators who produced the METR report in 6 days on-site; my understanding is that he did most of the technical heavy lifting, and that very few others are positioned to be able to do this. Probability of loss of control / takeover events. Conjunctions of bad actions by a model now do not seem substantially less likely than individual bad actions. Additionally, I read more about the rapid pace of bio capability improvements (eg EVO2 producing novel viruses to kill E. coli bacteria in a lab) relative to the weak state of defence mechanisms and ease of circumventing security to produce DNA sequences for lethal viruses, and current reliance on tacit knowledge as the barrier to bioweapons. I am concerned that an analogous cyber moment applied to biology is both plausible (>20%) and that under the current state of the field we are not on track to manage the ri