Ramez Naam: Can AI self-improvement overcome diminishing returns?

This is a linkpost for a new essay by futurist and sci fi author Ramez Naam: Can AI self-improvement overcome diminishing returns?

AI is already helping improve itself. The question is whether even fully autonomous recursive self-improvement (RSI) would cause a runaway intelligence explosion.The theory is that each generation of AI could build a better successor, faster than the last generation did. That could lead to a “fast takeoff,” with capabilities surging to artificial superintelligence (ASI) in a year, months, or even days.Here’s my take: Given our best current data, the AI self-improvement loop would need to be roughly 5–10× stronger to sustain itself, let alone run away. I’ll explain this math in section 8. I expect incredibly rapid AI progress by the standards of nearly any other technology. But the evidence we have doesn’t suggest a sudden explosion to incomprehensible superintelligence anytime soon.I could be wrong. Forecasters have repeatedly underestimated AI progress! I could well be next. One thing that’s clear is that we need better data. For now, let’s work with what we can measure, and stay open to breakthroughs that could change the picture.

AI is already helping improve itself. The question is whether even fully autonomous recursive self-improvement (RSI) would cause a runaway intelligence explosion.

The theory is that each generation of AI could build a better successor, faster than the last generation did. That could lead to a “fast takeoff,” with capabilities surging to artificial superintelligence (ASI) in a year, months, or even days.

Here’s my take: Given our best current data, the AI self-improvement loop would need to be roughly 5–10× stronger to sustain itself, let alone run away. I’ll explain this math in section 8. I expect incredibly rapid AI progress by the standards of nearly any other technology. But the evidence we have doesn’t suggest a sudden explosion to incomprehensible superintelligence anytime soon.

I could be wrong. Forecasters have repeatedly underestimated AI progress! I could well be next. One thing that’s clear is that we need better data. For now, let’s work with what we can measure, and stay open to breakthroughs that could change the picture.

(Note that this is the same essay as 'Where’s the “intelligence explosion”?' on Noah Smith's blog)