aiqbench takeAI Safety

This analysis clarifies that the likelihood of a software intelligence explosion depends on algorithmic efficiency or hardware scaling outpacing exponential growth, rather than mere increases in physical compute capacity. The argument demonstrates mathematically why standard assumptions about exponentially growing chip stocks do not inherently accelerate an intelligence singularity without super-exponential drivers.

Sources coveringLessWrong, Fortune

What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Compute growth doesn't make a software intelligence explosion more likely

LessWrongJuly 29, 20265 min read

This is a nitpick of a claim that I sometimes see in discussions of a software intelligence explosion. There is a careful analysis of the conditions under which an SIE will occur, assuming a fixed stock of compute. Then the argument concludes with "And in reality, compute is not fixed and is in fact growing exponentially, so an SIE is even more likely than this analysis suggests." This conclusion is false. Exponentially growing physical compute has no effect on the likelihood of an SIE. To see this, let mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-mn { display: inline-block; text-align: left; }

This conclusion is false. Exponentially growing physical compute has no effect on the likelihood of an SIE. To see this, let

denote effective compute, where is physical compute and is algorithmic efficiency. Better algorithms allow a fixed stock of hardware to perform as much work as a larger stock of physical compute – the SIE channel. But what does the physical compute channel do?

A necessary condition for an SIE is – the growth rate of effective compute is itself growing. [1] This means an SIE can happen only if

An exponentially growing compute stock means – but it also means is constant, so . As a result, exponential compute growth contributes nothing to the SIE. The necessary condition remains , in which physical compute doesn't appear.

What's going on here? Intuitively, an SIE requires super-exponential growth in , which requires either or to grow super-exponentially. The standard SIE analysis posits conditions for to grow super-exponentially, based on the feedback loop from self-improving AI. But if is growing super-exponentially, then will grow super-exponentially even if is fixed – so growth in is redundant. On the other hand, if only grows exponentially, exponential growth in can't push it into super-exponential territory. The product of two exponentials is still exponential.

The only way for to contribute to an intelligence explosion is if it grows super-exponentially, in a chip-production/chip technology feedback loop – but this is a much stronger condition than any required for an SIE.

Of course, growth in physical compute still raises the rate of AI progress – so it still shortens doubling times for any measure of AI capabilities. But it doesn't move the needle on the specific question of how likely is a software-only intelligence explosion.

This is not a sufficient condition – the growth rate also needs to be fast enough to outrun diminishing returns to research, which is the condition from Eth and Davidson (2025) – but its necessity is enough for our purposes. ↩︎

Full text under the publisher's open licence · 436 words

Read at LessWrong ↗Ask about this article