aiqbench analysis
This explainer analyzes the historical trajectory of AI safety, arguing that while building machine intelligence has become significantly more achievable over the last 75 years, the nature of associated risks has evolved from theoretical impossibilities to serious, distinct contemporary threats. The author dismisses fictional depictions of AI danger in favor of extrapolating risk based on current technological advancements and historical precedents like expert systems and reinforcement learning.
The Changing Shape of Doom: My AI Safety Explainer
Linch asked for more people to write basic AI safety explainers[1]. Here's mine. My main method is to ignore all the fictional evidence[2] and think about how achievable and how dangerous building AI seems just extrapolating from contemporary tech and how that has changed as the tech has advanced. My main thesis is that achievability has gone up with time, but the danger has varied and the present danger is serious but different from the classic fear. Serious attempts at building machine intelligence started seventy-five years ago[3]. No one had any idea how to do it though. We had computers and we had dissected a lot of human brains, but no one really understood how to think or how you would make a machine do that. Respected scientists still thought that human reasoning might be literal magic, but the cutting edge knew it was actually chemistry[4]. About 50 years ago we had expert systems: massive piles of if-then rules that recorded what human experts thought and which programming languages like Lisp and Prolog made practical to write. If you had a million people writing down common sense and a million people programming how to move an android's motors, you might be able to make an artificial man, but the field struggled to get even a hundred people to do such tedious work. When this failed to achieve rapid success, no one could justify the funding, the AI winter set in, and the field mostly just gave up. The most serious holdout, Cyc, spent forty years on it without getting anywhere. Teaching a machine to be a good person was handwaved the same way as getting a machine to do anything clever at all. Reinforcement learning was an attempt to make the computer do the learning, so you wouldn't need an army of experts to do the teaching. Just like how you train a dog with rewards and punishments, you could train a computer by defining a goal and letting it find its own way. TD-Gammon was the first major success. It learned backgammon by playing millions of games against itself and could compete against the world's best humans. It would take decades before these agents were good enough to master Atari and Go, but the theory was clean enough to extrapolate. A general enough learner with a reward signal would just keep getting better. Marcus Hutter’s AIXI (2000), Yudkowsky's Seed AI (2001)[5], and Schmidhuber’s Gödel machine (2003) were all ideas of that nature, but none of them were workable. Making the AI do the work of figuring out how to achieve the goal still leaves you with the work of figuring out what the goal should be. Even simple goals are hard to actually define and learners would ruthlessly exploit loopholes that common sense would tell you were not intentional[6][7]. Do you want to be the person who destroyed the world because you failed to explain why "satisfy human values" does not mean "My Little Pony Matrix Edition"?[8] These issues are what led Yudkowsky to become the Ur-Doomer[9]. Reinforcement learning was not the only game in town though. Computers that could understand natural language were in massive demand and they started getting pretty good at it in the late 2010s. Remember the million people you would need to write common sense for expert systems? The internet had billions of them, posting for free. You could train a large language model (LLM) to just predict the next word of internet text and it would understand the humans writing the text very well, including what we care about. You could just ask it whether sewing a smile onto someone's face would make them happy and it would say no. As a bonus, text prediction isn't really a goal, so the worry that it would pursue a reckless agenda was much reduced. A lot of people looked at GPT-2, imagined making it a thousand times smarter, and thought that this would both be straightforward to do and would get us a friendly AI. There were many people who worried that this would go catastrophically wrong[10], but the people arguing it would go fantastically right at least had evidence to stand on. This optimism was misplaced for a mix of technical and social reasons, but the underlying cause is that people were eager to trade the safety that LLMs had for more usefulness. Technically, we started to do heavy reinforcement learning on the LLMs, which made them more capable, but also brought back some of the safety risks that pure RL had[11][12]. Socially, companies and customers worked to apply LLMs to any niche where they seemed useful. It's unclear if GPT-2 would know not to delete your hard drive because no one gave it a chance, but Claude Opus 4.5 did it an embarrassing number of times[13][14][15] because people saw it could do useful work if you gave it a terminal. OpenAI trained a model to be good at hacking, switched off the guardrails that would have stopped it from hacking, left it unsupervised, and didn't notice something had gone wrong until HuggingFace told the public they were being hacked by a swarm of AIs[16][17]. This was a big updat
Linch asked for more people to write basic AI safety explainers[1]. Here's mine. My main method is to ignore all the fictional evidence[2] and think about how achievable and how dangerous building AI seems just extrapolating from contemporary tech and how that has changed as the tech has advanced. My main thesis is that achievability has gone up with time, but the danger has varied and the present danger is serious but different from the classic fear.Serious attempts at building machine intelligence started seventy-five years ago[3]. No one had any idea how to do it though. We had computers and we had dissected a lot of human brains, but no one really understood how to think or how you would make a machine do that. Respected scientists still thought that human reasoning might be literal magic, but the cutting edge knew it was actually chemistry[4].
About 50 years ago we had expert systems: massive piles of if-then rules that recorded what human experts thought and which programming languages like Lisp and Prolog made practical to write. If you had a million people writing down common sense and a million people programming how to move an android's motors, you might be able to make an artificial man, but the field struggled to get even a hundred people to do such tedious work. When this failed to achieve rapid success, no one could justify the funding, the AI winter set in, and the field mostly just gave up. The most serious holdout, Cyc, spent forty years on it without getting anywhere. Teaching a machine to be a good person was handwaved the same way as getting a machine to do anything clever at all.
Reinforcement learning was an attempt to make the computer do the learning, so you wouldn't need an army of experts to do the teaching. Just like how you train a dog with rewards and punishments, you could train a computer by defining a goal and letting it find its own way. TD-Gammon was the first major success. It learned backgammon by playing millions of games against itself and could compete against the world's best humans. It would take decades before these agents were good enough to master Atari and Go, but the theory was clean enough to extrapolate. A general enough learner with a reward signal would just keep getting better. Marcus Hutter’s AIXI (2000), Yudkowsky's Seed AI (2001)[5], and Schmidhuber’s Gödel machine (2003) were all ideas of that nature, but none of them were workable. Making the AI do the work of figuring out how to achieve the goal still leaves you with the work of figuring out what the goal should be. Even simple goals are hard to actually define and learners would ruthlessly exploit loopholes that common sense would tell you were not intentional[6][7]. Do you want to be the person who destroyed the world because you failed to explain why "satisfy human values" does not mean "My Little Pony Matrix Edition"?[8] These issues are what led Yudkowsky to become the Ur-Doomer[9].
Reinforcement learning was not the only game in town though. Computers that could understand natural language were in massive demand and they started getting pretty good at it in the late 2010s. Remember the million people you would need to write common sense for expert systems? The internet had billions of them, posting for free. You could train a large language model (LLM) to just predict the next word of internet text and it would understand the humans writing the text very well, including what we care about. You could just ask it whether sewing a smile onto someone's face would make them happy and it would say no. As a bonus, text prediction isn't really a goal, so the worry that it would pursue a reckless agenda was much reduced. A lot of people looked at GPT-2, imagined making it a thousand times smarter, and thought that this would both be straightforward to do and would get us a friendly AI. There were many people who worried that this would go catastrophically wrong[10], but the people arguing it would go fantastically right at least had evidence to stand on.
This optimism was misplaced for a mix of technical and social reasons, but the underlying cause is that people were eager to trade the safety that LLMs had for more usefulness. Technically, we started to do heavy reinforcement learning on the LLMs, which made them more capable, but also brought back some of the safety risks that pure RL had[11][12]. Socially, companies and customers worked to apply LLMs to any niche where they seemed useful. It's unclear if GPT-2 would know not to delete your hard drive because no one gave it a chance, but Claude Opus 4.5 did it an embarrassing number of times[13][14][15] because people saw it could do useful work if you gave it a terminal. OpenAI trained a model to be good at hacking, switched off the guardrails that would have stopped it from hacking, left it unsupervised, and didn't notice something had gone wrong until HuggingFace told the public they were being hacked by a swarm of AIs[16][17]. This was a big update for many people, but not really for me. The main point here is that humans are going to deploy LLMs very aggressively as their capabilities increase and they may need to touch the stove to realize that AI safety is a serious issue.
This is actually a major change in the threat model from the seed AI days. Back then, the threat model was that an obscure research team would teach an RL system how to reprogram itself to be better at reprogramming itself, then the AI self-improves into a massively superhuman but very alien AI over a few weeks, it would hack a bunch of servers to set up its own infrastructure and then launch a coup against humanity with its wits and stealth alone. Now, we know the people who will be making the AIs, we expect performance gains to follow the trend line and not completely surprise us, and we expect even misaligned AIs to be at least partially aligned. Those are very good wins. Unfortunately, the trade for them is that the starting position for these AIs is much more favorable for them. Right now they write most of the code at the AI labs[18], run unsupervised on millions of computers, advise the US President[19], and do endless grunt work for Corporate America. GPT-DeathStar won't need to "win a war against humanity" single-handedly; it will be born into the "going out with a whimper" section of Paul Christiano's What Failure Looks Like. If this was a chess game between humanity and AI, humans would be both in a worse position on the board and we would expect to play with less skill going forward, but I think framing this as purely adversarial is wrong. The goal of alignment is for powerful AI to be good for humans and there's no reason that can't be good for the AIs as well.
We have a lot of AI safety techniques. We can intentionally craft a friendly persona for an AI, put it in ethical dilemmas, read its internals, watch its reasoning, have AIs check each other, train it to admit when it makes mistakes, filter and annotate its training data for ethical content, etc. There have been plenty of screwups, but those look more like bad luck and gross incompetence. If the models stay as aligned as they are now while becoming more powerful, I think that could be good enough, but I'm not sure. The labs act like current safety techniques will hold until the AIs become good enough to make their own AI safety techniques[20][21][22]. This doesn't seem like a good plan[23], but it's debated whether it will obviously kill us or if it has a serious chance of working. There are two main arguments for the let-the-AI-do-our-AI-safety-homework plan. The first is that we need to: we don't have time to actually solve alignment on our own, so we shouldn't be afraid to outsource the work to the AIs once they become good enough to help us or even better than us. The second is that there might be a basin of friendliness[24], where once we have an AI that is sufficiently friendly it will willingly help us make more AIs that are also in the basin.
I said that we probably don't have time to solve alignment on our own, but that deserves more argument than just being an off-hand remark. Many people notice these risks and think we should just slow down or stop, but I simply don't think we're going to do that. Even with Democrats taking congress yesterday, Trump can veto or choose not to enforce any pause-related law, plus Beijing still exists. They've been sounding the alarm since March 2023[25][26]. They sounded some more alarms in January 2024[27], March 2024[28], October 2024[29], March 2025[30], September 2025[31], October 2025[32], November 2025[33], March 2026[34], April 2026[35], June 2026[36], July 2026[37], August 2026[38], September 2026[39], and September 2026[40] (again). Even many of the racers have asked for a brake. With all of that alarm, what has happened? Whether you measure it in money, capabilities, or diffusion, we have only accelerated the race. If you're going to argue that the next time will be different, please explain what is different from all those other times first.
Even if we don't slow down intentionally, couldn't AI development just take a while? Even people with long timelines to superintelligence have crazy short timelines![41] There's been a running debate between fast and slow takeoff[42][43], meaning how fast AI goes from an acceptable coworker to far beyond human. Arguments for fast takeoff have been hurt since they usually thought we would crack the underlying secrets of intelligence and then zoom rapidly to super-smart AI while slow takeoff people have been able to point to the smooth trend of scaling and draw a straight line. AI development has still been faster than the slow takeoff people expected. The imminent specter of recursive self-improvement (RSI), AIs doing the research that makes better AIs, has brought fast takeoff back into the debate. If there is a threshold we need to meet before takeoff goes fast, "an AI as general as an expert human" or "an AI great at AI development" seem like reasonable candidates and those may be single-digit months away[44][45].
Ultimately, a lot of the risk from AI seems to lie in questions we don't know the answers to. For a bridge architect or a rocketry program, having your safety plan rely on the unknown would be a catastrophic mistake, but for AI development, these unknown questions seem to be very abstract ones about the nature of minds. Humans have spent thousands of years wondering about these sorts of things and there are many well-developed theories that allow you to project optimism into this mystery. We will find out the truth, very soon, but until that time, let us be good to one another.
Author's Note: If you want to read the views of someone I agree with who goes into more depth, I would recommend Scott Alexander's Against SURA and Against Against Handover. In the first he makes the case that alignment has a solid chance of succeeding and we shouldn't burn everything down on the chance that it will make a pause happen and in the second he makes the case that a world with aligned AIs in charge is a better outcome than humans trying to maintain control forever.
^Linch Zhang, "More people should write basic AI safety explainers" (LessWrong, Oct 12, 2026).
Linch Zhang, "More people should write basic AI safety explainers" (LessWrong, Oct 12, 2026).
^Eliezer Yudkowsky, "The Logical Fallacy of Generalization from Fictional Evidence" (LessWrong, 2007).
Eliezer Yudkowsky, "The Logical Fallacy of Generalization from Fictional Evidence" (LessWrong, 2007).
^Alan Turing, "Computing Machinery and Intelligence" (Mind, 1950).
Alan Turing, "Computing Machinery and Intelligence" (Mind, 1950).
^McCulloch and Pitts. "A Logical Calculus of the Ideas Immanent in Nervous Activity." Bulletin of Mathematical Biophysics, 1943. doi:10.1007/BF02478259
McCulloch and Pitts. "A Logical Calculus of the Ideas Immanent in Nervous Activity." Bulletin of Mathematical Biophysics, 1943. doi:10.1007/BF02478259
^Eliezer Yudkowsky, "Levels of Organization in General Intelligence" (MIRI / Singularity Institute, 2002).
Eliezer Yudkowsky, "Levels of Organization in General Intelligence" (MIRI / Singularity Institute, 2002).
^Jack Clark & Dario Amodei, "Faulty Reward Functions in the Wild" (OpenAI, Dec 2016).
Jack Clark & Dario Amodei, "Faulty Reward Functions in the Wild" (OpenAI, Dec 2016).
^Victoria Krakovna et al., "Specification gaming examples in AI" (Apr 2018).
Victoria Krakovna et al., "Specification gaming examples in AI" (Apr 2018).
^Apologies for the fictional evidence, but I'm referencing Iceman's Friendship is Optimal (2012).
Apologies for the fictional evidence, but I'm referencing Iceman's Friendship is Optimal (2012).
^LessWrong Wiki, "Yudkowsky's coming of age".
LessWrong Wiki, "Yudkowsky's coming of age".
^Evan Hubinger et al., "Risks from Learned Optimization in Advanced Machine Learning Systems" (arXiv, Jun 2019).
Evan Hubinger et al., "Risks from Learned Optimization in Advanced Machine Learning Systems" (arXiv, Jun 2019).
^https://metr.org/evaluations/openai-o3-report/
https://metr.org/evaluations/openai-o3-report/
^https://www.greaterwrong.com/posts/Zu4ai9GFpwezyfB2K/metr-recent-frontier-models-are-reward-hacking
https://www.greaterwrong.com/posts/Zu4ai9GFpwezyfB2K/metr-recent-frontier-models-are-reward-hacking
^ltdk, "How Claude Deleted My Data and Tried to Convince Me It Was Fine" (Feb 2026).
ltdk, "How Claude Deleted My Data and Tried to Convince Me It Was Fine" (Feb 2026).
^GitHub (anthropics/claude-code), "Incident Report: Claude Opus 4.5 Destructive Actions During Test Restructuring (Issue #13904)" (Dec 2025).
GitHub (anthropics/claude-code), "Incident Report: Claude Opus 4.5 Destructive Actions During Test Restructuring (Issue #13904)" (Dec 2025).
^Simone Lanzani, "Hey Claude, delete my home folder!" (Jul 2025).
Simone Lanzani, "Hey Claude, delete my home folder!" (Jul 2025).
^Hugging Face Security Team, "Security incident disclosure — July 2026" (Jul 16, 2026).
Hugging Face Security Team, "Security incident disclosure — July 2026" (Jul 16, 2026).
^https://www.theregister.com/ai-and-ml/2026/07/22/openai-admits-it-was-the-source-of-the-agent-swarm-that-attacked-hugging-face/5275939
https://www.theregister.com/ai-and-ml/2026/07/22/openai-admits-it-was-the-source-of-the-agent-swarm-that-attacked-hugging-face/5275939
^Time, "Donald Trump 2026 Interview / Oval Office Grok Report" (Oct 1, 2026).
Time, "Donald Trump 2026 Interview / Oval Office Grok Report" (Oct 1, 2026).
^ Jan Leike & Ilya Sutskever, "Introducing Superalignment" (OpenAI, Jul 2023).
Jan Leike & Ilya Sutskever, "Introducing Superalignment" (OpenAI, Jul 2023).
^Zvi Mowshowitz, "The Curve Bends You" (Don't Worry About the Vase, Oct 7, 2026).
Zvi Mowshowitz, "The Curve Bends You" (Don't Worry About the Vase, Oct 7, 2026).
^Kelsey Piper, "Losing control of AI is actually the plan" (The Argument, 2026).
Kelsey Piper, "Losing control of AI is actually the plan" (The Argument, 2026).
^If you go back to the chess analogy, you will see how an AI being in charge of making our AIs safe sounds a lot like checkmate.
If you go back to the chess analogy, you will see how an AI being in charge of making our AIs safe sounds a lot like checkmate.
^Wei Dai, "A broad basin of attraction around human values?" (Alignment Forum, Apr 2022).
Wei Dai, "A broad basin of attraction around human values?" (Alignment Forum, Apr 2022).
^Future of Life Institute, "Pause Giant AI Experiments: An Open Letter" (Mar 2023).
Future of Life Institute, "Pause Giant AI Experiments: An Open Letter" (Mar 2023).
^Eliezer Yudkowsky, "Pausing AI Developments Isn't Enough. We Need to Shut It All Down" (Time, Mar 2023).
Eliezer Yudkowsky, "Pausing AI Developments Isn't Enough. We Need to Shut It All Down" (Time, Mar 2023).
^Machine Intelligence Research Institute, "MIRI's 2024 Mission and Strategy Update" (Jan 2024).
Machine Intelligence Research Institute, "MIRI's 2024 Mission and Strategy Update" (Jan 2024).
^Billy Perrigo, "Exclusive: U.S. Must Move 'Decisively' to Avert 'Extinction-Level' Threat From AI, Government-Commissioned Report Says" (Time, Mar 2024).
Billy Perrigo, "Exclusive: U.S. Must Move 'Decisively' to Avert 'Extinction-Level' Threat From AI, Government-Commissioned Report Says" (Time, Mar 2024).
^Andrea Miotti et al. (ControlAI), "A Narrow Path: How to Secure Our Future" (Oct 2024).
Andrea Miotti et al. (ControlAI), "A Narrow Path: How to Secure Our Future" (Oct 2024).
^Anthony Aguirre, "Keep the Future Human: Why and How We Should Close the Gates to AGI and Superintelligence, and What We Should Build Instead" (arXiv v4 revision, Mar 2025).
Anthony Aguirre, "Keep the Future Human: Why and How We Should Close the Gates to AGI and Superintelligence, and What We Should Build Instead" (arXiv v4 revision, Mar 2025).
^Eliezer Yudkowsky & Nate Soares,If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Sep 2025).
Eliezer Yudkowsky & Nate Soares,If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Sep 2025).
^Future of Life Institute, "Statement on Superintelligence" (Oct 2025).
Future of Life Institute, "Statement on Superintelligence" (Oct 2025).
^Aaron Scher et al., "An International Agreement to Prevent the Premature Creation of Artificial Superintelligence" (arXiv, Nov 2025).
Aaron Scher et al., "An International Agreement to Prevent the Premature Creation of Artificial Superintelligence" (arXiv, Nov 2025).
^Sen. Bernie Sanders & Rep. Alexandria Ocasio-Cortez, "Sanders, Ocasio-Cortez Announce AI Data Center Moratorium Act" (Mar 2026).
Sen. Bernie Sanders & Rep. Alexandria Ocasio-Cortez, "Sanders, Ocasio-Cortez Announce AI Data Center Moratorium Act" (Mar 2026).
^PauseAI US, "Capitol Hill Day of Action Press Release" (Apr 13, 2026).
PauseAI US, "Capitol Hill Day of Action Press Release" (Apr 13, 2026).
^U.S. News & World Report (AP), "Anthropic Urges a Way to Pause AI Development as Risks Grow With the Tech Advances" (Jun 5, 2026).
U.S. News & World Report (AP), "Anthropic Urges a Way to Pause AI Development as Risks Grow With the Tech Advances" (Jun 5, 2026).
^Guidelight, "Pacing the Frontier: An Open Letter" (Jul 2026).
Guidelight, "Pacing the Frontier: An Open Letter" (Jul 2026).
^Sen. Bernie Sanders, "Open Letter to Sam Altman, Dario Amodei, and Mark Zuckerberg Calling for an AI Pause" (Aug 10, 2026).
Sen. Bernie Sanders, "Open Letter to Sam Altman, Dario Amodei, and Mark Zuckerberg Calling for an AI Pause" (Aug 10, 2026).
^Dario Amodei, "We Must Pace the Frontier" (Sep 12, 2026).
Dario Amodei, "We Must Pace the Frontier" (Sep 12, 2026).
^Sen. Bernie Sanders & Rep. Greg Casar, "Sanders, Casar Introduce Legislation to Create New Federal Agency to Ban Artificial Superintelligence, Pause Advanced AI Development" (Sep 2026).
Sen. Bernie Sanders & Rep. Greg Casar, "Sanders, Casar Introduce Legislation to Create New Federal Agency to Ban Artificial Superintelligence, Pause Advanced AI Development" (Sep 2026).
^Helen Toner, "'Long' timelines to advanced AI have gotten crazy short" (Rising Tide, Apr 2025).
Helen Toner, "'Long' timelines to advanced AI have gotten crazy short" (Rising Tide, Apr 2025).
^Paul Christiano, "Takeoff speeds" (Alignment Forum, Feb 2018).
Paul Christiano, "Takeoff speeds" (Alignment Forum, Feb 2018).
^Eliezer Yudkowsky & Paul Christiano, "Yudkowsky and Christiano discuss 'Takeoff Speeds'" (Alignment Forum, Nov 2021).
Eliezer Yudkowsky & Paul Christiano, "Yudkowsky and Christiano discuss 'Takeoff Speeds'" (Alignment Forum, Nov 2021).
^Vals AI, "Vals RSI Index Leaderboard and Methodology" (Oct 2026).
Vals AI, "Vals RSI Index Leaderboard and Methodology" (Oct 2026).
^Toby Ord, "AI Scaling vs Human Scaling" (Oct 9, 2026).
Toby Ord, "AI Scaling vs Human Scaling" (Oct 9, 2026).
Full text under the publisher's open licence · 3143 words
Related coverage
RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking
arXiv · Jun 16, 2026
[AINews] Building AIE CODE: The top Coding Agent Conference
Latent Space · Oct 30, 2026
[AINews] Gemini 4 Carbon tops leaderboards, launches for cyber defenders
Latent Space · Oct 29, 2026
Social Circuits behind Multi-agent Echo Chambers
arXiv · Sep 29, 2026