Robustness: Backfire Risks in AI Safety and Governance

Published on October 15, 2026 6:31 AM GMT A guide to unintended consequences of AI safety interventions and how you can mitigate them. Introduction My cofounder and I are launching Impartial Projects, a studio for AI safety field strategy. We hope to start the projects most urgently needed in the AI safety ecosystem (which we understand to include AI governance). We think that the founding of new organizations (as well as increasing management capacity) are among the most effective things we can do at the moment to expand the surface area of AI safety. Surface area, it seems to me, is currently the scarcest resource. Another benefit we want to provide is our particular attention to robustly positive theories of change. We prefer to take a somewhat risk‑averse approach to AI safety, whether out of literal risk aversion or because we assign a higher probability to s‑risks than most. Interventions in AI safety are notorious for the backfire risks they harbor. Of course, that does not mean one should turn one’s back on these interventions altogether. Rather, it means that, depending on the intervention in question, more monitoring and mitigation methods are needed to minimize the various backfire risks that are part of the intervention’s risk profile. It is critical to go into this work with eyes wide open, cognizant of its benefits and risks. It’s critical to strike the right balance here. With my last startup, I made the mistake of erring too much on the side of caution and putting the greater part of a year into strategizing around risks and mitigations that, in the end, never became necessary. After all, delay and inaction are (risky) actions of their own right. Arguably, any writing about risks is an infohazard because it will be read predominantly by people who are already too paralyzed by risks. I later created a model that can help you decide how much time is rational for you to invest in this work, given your background assumptions. I hope this catalog will help founders navigate their particular intervention areas with greater confidence and foresight. The catalog is organized as a 2-D matrix with the axes: Mechanism. How the backfire happens. Fifteen common mechanisms, plus one cross-cutting modifier. Actor. Who adapts to the intervention in a way that produces the backfire. These two dimensions are selected to maximize their orthogonality. Although it did not work out perfectly, by and large they produce a matrix that can now serve as a guide for us to notice underexplored areas. That is to say, almost every actor can be subject to almost every mechanism, but some actor‑mechanism combinations have been explored to a greater extent than others, and some not at all, to my knowledge. There are more dimensions with questionable orthogonality, such as the resulting harm of the backfire, its probability and severity, the locus of the intervention, the time horizon or urgency, and what we can do to monitor and mitigate these risks. Note that, for simplicity, I’ve limited myself to short-term, causal risks, i.e. ignored implications for long-term space governance, what our behavior tells us about the universal prior or civilizations beyond the accessible universe, etc. How to Read an Entry Harms. The harms associated with the backfire, covering x-risks, s-risks, inter‑AI conflict, great-power conflict, concentration of power, epistemic degradation, and digital suffering. Locus. The relevant venue: training, evaluation, deployment, infrastructure, governance, institutions, research publication, discourse. Horizon. The urgency of mitigations: Now means the dynamic is already observable. Transition refers to the time when the models cross the human level in relevant domains. Post-transition refers to post-AGI lock-in effects. Entries marked urgent are those where the dynamic is well underway or where our design decisions will be hard to reverse. Severity and probability. Severity is how bad the backfire would be if it materializes (broad categories: low, medium, high, extreme). Probability is the chance that the backfire materializes to a meaningful degree if we pursue the intervention in roughly its present form (low, medium, high). Rough judgment calls, low confidence. Monitor. Warning signs that the backfire is happening. Mitigate. Design changes that reduce the backfire risk. The Two Taxonomies Mechanisms Selection effects and Goodharting. A proxy for safety becomes an optimization target. The problem with that is not so much the wasted effort that goes into optimizing something we don’t terminally care about, but the adversarialness of the relationship and how it creates a separate optimization pressure to increasingly obfuscate the safety that we would actually prefer to measure. Dual use and capability externality. Increases in safety (think RLHF), often come with increases in capabilities that can accelerate further development – be it through recursive self-improvement or through securing more inv