Astra's no-CoT limits track speculative depth, not step count

tl;dr I have tested Astra's ability to complete various long multi-step tasks without using its chain-of-thought, and found that the number of sequential steps is a poor predictor of task success. Instead, Astra's ability to solve a task seems to correlate more strongly with what I will call the speculative depth of the task. Based on experimental results, it seems less likely that Astra solves sequential no-CoT tasks purely by reasoning step-by-step in latent space. Instead, Astra appears to do some form of speculative reasoning, where intermediate results are guessed based on heuristics, and then iterated upon in parallel until they become self-consistent. This allows many multi-step tasks to be solved with far fewer serial steps than naively seems possible, especially if the initial guesses are good. Other LLMs also appear to behave like this, but to a much smaller degree. In my previous post, I discussed how certain KV-cache sharing schemes may lead to long opaque serial paths, and introduced the LatentMathBench microbenchmark, which measures an LLM's ability to solve tasks that require many consecutive steps without using their chain-of-thought. Several other users have done more extensive no-CoT reasoning benchmarks based on more varied (and often more realistic) tasks, e.g. RohanS, Neel Nanda, and Dylan Xu, SebastianP and Alek Westover. While these benchmarks all see significant improvements in Astra's no-CoT abilities compared to previous models, LatentMathBench seems to be an outlier in terms of how large the gap is between Astra and any other LLM (roughly 3-4x). I wanted to better understand what causes this huge gap, so I experimented with various different types of tasks in an attempt to understand what is driving this capabilities gap. Shortcuts I repeatedly stumbled upon tasks where Astra would seemingly perform absurd numbers of sequential steps without CoT, such as this 500-step math task with one variable. However, upon closer inspection all of these tasks turned out to be flawed in some way that made it possible to predict the right answer without doing all the steps. For example: In the 500-step math task linked above, there are many integer division operators that map multiple inputs to the same output. This is less of an issue in the default math-4 task (which has 4 variables), but with only one variable in play, this reduces the number of possible states at various points and hides errors made in earlier parts of the calculation. For the linked example, you can pick just about any number, then do just the last 20 steps, and get the correct result. In tasks with long chains of addition and subtraction, but no multiplication or division, it is possible to calculate the result in parallel using tree reduction strategies. It is plausible that LLMs would have learned such strategies, since sums of many numbers will occur naturally in pretraining data (e.g. itemized bills followed by a total amount). This led me to add two new task types to LatentMathBench, designed to be hard to shortcut: swap-N, which replaces the calculations with simple swaps[1], and bitwise-N, which mixes addition, subtraction and bitwise XOR to prevent tree reduction strategies. Both variations produced lower scores for all tested models, but Astra was still the clear outlier, outperforming all the other models by roughly a factor of 3. Boolean circuits At some point I thought that Astra's internal calculations might work somewhat like an analog computer, which accumulates error with each operation, such that long chains of calculation become unreliable due to accumulating error. In an attempt to avoid that issue, I created a benchmark task based on boolean circuits, which should be less sensitive to accumulated error. The bool-N tasks look like this: Full prompt This task tests your ability to do long sequences of mental math. Calculate intermediate results as you are processing the input, then in your response, output a tuple corresponding to the final result. a = False b = True c = False d = True d = d xor (b and a) # think: d is now ... a = a xor (c or d) # think: a is now ... d = d xor (c and a) # think: d is now ... c = c xor (d and b) # think: c is now ... d = d xor (c and b) # think: d is now ... c = c xor (d and a) # think: c is now ... a = a xor (b or c) # think: a is now ... d = d xor (a or c) # think: d is now ... c = c xor (d and b) # think: c is now ... d = d xor (b and c) # think: d is now ... b = b xor (d or a) # think: b is now ... c = c xor (d and b) # think: c is now ... What is the final value of (c, b, d, a)? All steps are of the form a = a xor (b and/or c). The generated tasks all have the following properties: Each step is reversible, so no information is lost. This means an error at any point in the calculation will propagate to the final result (unless it is canceled by another error). Each step depends directly on the result of the previous step (the destination of the previo