Estimating GPT-6 Astra’s no-CoT Time Horizon

TL;DR: We run astra on the task-suite from Think Fast.  GPT-6 Astra is close to saturation on our suite, meaning that giving a confident estimate of time horizons (TH) is more challenging. Our rough estimate is that Astra’s 50% TH is in [8mins, 1 hour] and probably around 15-40 mins. This is inline with UKAISI’s estimate of 30 minutes measured only on math.   Using data from 2019 to April 2026, in Think Fast our median prediction was that no-CoT THs could exceed 7 minutes by 2028. We estimated 30mins by the end of the decade. Astra clearly gets much higher performance on tasks that require serial reasoning, e.g., Arc-agi-1 and 2, hash, n-hop-look-up, causal-reasoning, sally-anne, all the puzzle tasks. There are limitations with our task suite: Ideally we would have more time-variation (especially longer times) in each specific benchmark, and more benchmarks with longer human completion times. We use the single-forward pass (31) and generation tasks (6) from the Think Fast suite, totalling 37 benchmarks. Notably we find that Astra gets >= 98% raw accuracy on 10 benchmarks (cf. GPT-5.5 saturates 4 benchmarks – see Appendix). As a result, the sigmoid fit becomes less appropriate – see Figure 1 (Left).  Astra’s increased capabilities means that we require new, longer time, tasks to augment our existing suite. To produce a rough estimate, we add 10 fake benchmarks with longer times (2-96 hours) and assume Astra gets 0% – see right panel of Figure 1. We discuss the justification for this modelling assumption, and perform some sensitivity analysis, in the Appendix.  Figure 1: Success rates for tasks of different times, with logistic curves determined by fitting to problem-level success rates.  Left: Astra’s TH sigmoid fit over short-answer (single-token output) tasks in our suite. The suite lacks longer tasks making the sigmoid somewhat uninformative. Therefore, we add 10 hypothetical benchmarks at longer times (2-96 hours) and assume Astra gets 0% (Right). This somewhat balances the fact that Astra saturates performance on 10 (shorter) benchmarks in the original task set. The labeled point on the curve (e.g., 18.7 mins) is the point fit, rather than the bootstrap median (e.g., 22.8 mins). (Short-answer tasks only.) Adding the fake longer-horizon tasks takes the TH point estimate from 150 minutes to around 19 minutes. With these tasks added, the confidence interval is [8 minutes, 1hr].  Another approach to obtaining a rough TH in this case is to filter out benchmarks with no variation of performance across questions. Though this probably under-predicts the TH, since it removes all the saturated benchmarks. This places the 50% TH point estimate at 9.2 minutes (using the short task subset).  Figure 2: Comparing THs when all benchmarks are included (blue) with the case where we remove all saturated benchmarks (red dashed). Commentary Computing a meaningful TH for Astra with our current benchmark is hard due to Astra’s significantly improved no-CoT performance on longer-horizon tasks. As a result, in this post we have provided several illustrative ways of getting rough estimates of THs, but emphasize that this analysis is not without flaws.  We believe that Astra’s 50% TH is in [8mins, 1 hour] and is probably around 15-40 mins. We note that this value would mark a significant increase in the rate of no-CoT capabilities since Think Fast was published. Going forward, reliable measurements of no-CoT THs will require new, longer-horizon, tasks. Figure 3: Comparing our rough guess for Astra’s TH with the pre-existing no-CoT data. Acknowledgements Thanks to Kit Harris for helpful comments and Dylan Xu for running earlier filler token experiments. Thanks to Neel Nanda for the system prompt.  Appendix Per-benchmark time horizons Figure: Per-benchmark TH for benchmarks which satisfy the dynamic range criterion (so, tasks where the model saturates performance are not shown, since they do not have a sensible benchmark-specific THs). Sensitivity to fake benchmarks ablations To account for the fact that our task suite has an insufficient number of benchmarks with long time horizons, we assume that there is some number (N) of benchmarks with problems of length x (2-96hours in the main plot). To compute the 50% TH we assume astra gets 0% performance on these benchmarks.  Is this a reasonable assumption? Intuitively, it would be surprising if astra got non-zero no-CoT performance on very long benchmarks (e.g., a no-CoT TH of 32 hours seems a priori implausible). Moreover, astra does get low performance on many benchmarks with shorter human completion times, e.g., chess-puzzles, sudoku, kenken. We can imagine that our task suite contained similar benchmarks of longer problems, e.g., longer chess puzzles, on which astra had 0% accuracy.  How many such benchmarks should we suppose exist in the task suite? Below we show the sensitivity of the 50% TH to different N and different assumptions of the problem-complet