What is recursive self-improvement, and what would it mean to ban it?

Published on October 2, 2026 10:32 PM GMT This post was originally written for my Substack. Please feel free to subscribe if this kind of content interests you! Frontier AI companies have long had the stated goal of automating their own research. The default plan has been to build models that are even better than their most skilled researchers at advancing the frontier, and let these models build their own successors; a feedback loop that will ultimately result in superintelligence that dwarfs the collective intelligence of humanity. This process is known as recursive self-improvement (RSI). @jayelmnop on Twitter (May 2025) The problem with this plan is that it might run entirely out of human control, making it more likely that the resulting superintelligence will kill everyone. So as AI models are increasingly accelerating their own development, AI companies are starting to think that this RSI thing may not be such a good idea after all. Anthropic CEO Dario Amodei recently wrote on his blog that RSI “must be pursued very carefully, if at all”. OpenAI’s official blog states: “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely”. Similar calls are coming from elsewhere. Former DeepMind researcher Alex Turner wrote in The Guardian that companies must be prevented from “allowing AI to self-improve into an uncontrollable level of intelligence”. And New York Times columnist Ezra Klein dedicated an entire op-ed to calling for a ban on RSI: Human beings need to control the frontier, and controlling the frontier means stopping the labs from doing something they are on the cusp of doing: recursive self-improvement, the process by which A.I.s begin autonomously building and improving new generations of more powerful A.I.s at ever more rapid speeds. Sounds simple enough. We probably should just not let AI models build successive AI models in ways that human researchers can’t oversee, because that sounds very dicey. But there’s another wrinkle, which is that no one can agree on what RSI actually is. Sam Altman made this point in an interview with Fortune: “I think it’s very hard to say what a ban on RSI means”. Maybe he is just participating in the classic AGI Lab CEO practice of injecting undue slipperiness into the definition of various terms with the aim of confusing everyone, but he does have a point. There are many different definitions of RSI, which I’ll cover below. Even if we could agree on a definition, it is unclear to what extent each of them is already occurring, or whether we’ll know when it is. So “banning RSI” will require picking a somewhat arbitrary (and hopefully very conservative) threshold, and sticking with it. What even is RSI?   Here is every definition of RSI that I could come up with, listed from most to least expansive: #1 Any time a technology is used to help improve itself, this is RSI. This is the broadest possible way to define RSI, and I think a pretty useless one in our present situation. Under this definition, some version of RSI has been happening throughout the history of technological progress; a blacksmith can use his hammer, tongs, and anvil to help craft better hammers, tongs, and anvils. A more useful one would take into account the ways in which RSI in the AI context specifically will be qualitatively different; it will be faster, less easy to oversee, and less controllable. #2 If humans are using AI to help with their work in literally any way at all, this is RSI: This version of RSI is already well underway. For example, Anthropic reports that Claude currently “collaborates” in over 90% of work at the company and “leads” 26%. This is also the fairly conservative threshold at which Ezra Klein proposes we institute a ban: “A couple of years ago, none of these labs had turned substantial coding over to the A.I.s. It was human beings typing code at human speeds with our clumsy human fingers. Now, most of the code is written by AI. So as a first step, we could just go back to where none of the code is written by AI.” #3 If AI can replace a human researcher of a certain level, this is RSI: We can also think of RSI as a thing that happens once AI can replace a certain level researcher at an AI company. This need not result in a closed loop where the next generation of model is built without any human input; OpenAI claims to have internally developed an AI research “intern” that can “carry out well-defined research tasks under human direction”. They have the grander goal of developing a fully automated AI researcher by March 2028. This researcher, if successfully developed, would presumably cross the much harder threshold of choosing which research questions to pursue and judging success (the ever-elusive “research taste”). As of summer 2026, frontier models are apparently still quite bad at this. But now we’re stuck at another definitional crossroads: is it RSI if models are replacing junior researchers but still need the