How concerned should we be about Astra's recurrent architecture?

Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth. What architecture is Astra likely to have? The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across different token positions beyond what's passed in ordinary transformers. A longer discussion can be found in my past post. Looped transformers are not the scariest possible version of neuralese. In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory, and there is presumably some loop count beyond which additional processing of the same token will stop helping, so the maximum serial reasoning depth that can be practically achieved with this architecture is bounded. As we'll see below, OpenAI has likely further constrained the loop count below the practical maximum to make sure that Astra's serial depth isn't much larger than that of existing models. Nevertheless, it's a step toward a paradigm where more of the reasoning is opaque; the important question is how big that step is. How bad is this? Initially, the news seemed sharply at odds with OpenAI's commitment to preserve chain-of-thought monitorability. A few hours later, Jakub Pachocki from OpenAI soothed the worst fears, clarifying that the hidden serial depth of Astra is not substantially larger than that of GPT-4: Jakub Pachocki: I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program. Tomek Korbak, Mikita Balesni, and Micah Carroll soon made similar statements. This is consistent with The Information's article, which mentions that OpenAI is limiting the use of the technique in order to preserve a legible CoT. It also makes sense in light of OpenAI's alignment strategy, which continues to heavily rely on CoT monitorability.[1] As thebes argues, effective depth matters much more than the architectural details for CoT monitorability, and there's nothing inherently more difficult about monitoring a 32-layer model looped twice than a 64-layer model looped once. (In fact, I would personally guess that the former is slightly easier to monitor, since weight-tying constrains the expressivity.) However, one might reasonably worry that OpenAI has trained the model to use a large number of recurrent loops and simply constrained it to a small loop count during inference for now. The number of loops can then be viewed as a dial that can be turned up with trivial effort as soon as competitive pressures demand it. Even if OpenAI hasn't trained the model to use a larger number of loops, we might worry that OpenAI has set off a race to the bottom toward deeper and deeper looped transformers, and others will build such models in the future even if OpenAI doesn't. Ryan Greenblatt has expressed both concerns well: Ryan Greenblatt: Transparency about the opaque serial depth is great, but this statement is consistent with Astra having a configurable "dial" that is currently set to a low depth but could be trivially increased. We need more info to see how concerning these architectural changes are, including: Are there readily available ways to deploy this AI with much higher serial depth (that would be commensurately more performant)? This should include things like tiny amounts of fine-tuning to productively increase the number of iterations. Is the AI a large or above-tre