aiqbench analysis
The State Stream Transformer V2 introduces a novel architecture that leverages nonlinear recurrence within continuous latent spaces to enhance reasoning capabilities without discarding residual information. By enabling parallel training of this recurrence mechanism, the model achieves significant performance gains on complex benchmarks like GPQA-Diamond while maintaining parameter efficiency. This work represents a substantive advancement in architectural design for large language models, specifically targeting the optimization of latent reasoning processes.
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
arXiv:2605.00206v2 Announce Type: replace-cross Abstract: Current transformers discard their rich latent residual stream between positions, reconstructing latent reasoning context at each new position and leaving potential reasoning capacity untapped. The State Stream Transformer (SST) V2 enables parameter-efficient reasoning in continuous latent space through an FFN-driven nonlinear recurrence at each decoder layer, where latent states are streamed horizontally across the full sequence via a learned blend. This same mechanism supports continuous latent deliberation per position at inference time, dedicating additional FLOPs to exploring abstract reasoning before committing to a token. A two-pass parallel training procedure approximates the sequential recurrence, making co-training computationally practical. Hidden state analysis shows that the state stream facilitates reasoning through sharp, content-dependent reorganisations in continuous latent space; the LM head exposes the resulting latent belief states through the output distribution, while the state stream carries them forward to influence future positions. A learned probe shows that at the first generated token, the latent state already predicts whether the eventual answer will survive or break under additional latent computation for every subsequent position. Co-trained into an existing 27B backbone using only a small dataset of GSM8K examples and evaluated using an oracle to route each question to a depth of one to four recurrent forward passes per token, the SST achieves an architectural capacity bound of 61.11% on out-of-distribution GPQA-Diamond, a +15.15 point gain over a fine-tuning-matched baseline, and cuts that same baseline's remaining GSM8K errors by 46%. Together, these results provide a method for efficiently training a nonlinear recurrence and show that the state stream provides additional reasoning capacity beyond that of an otherwise training-matched transformer.
Abstract:Current transformers discard their rich latent residual stream between positions, reconstructing latent reasoning context at each new position and leaving potential reasoning capacity untapped. The State Stream Transformer (SST) V2 enables parameter-efficient reasoning in continuous latent space through an FFN-driven nonlinear recurrence at each decoder layer, where latent states are streamed horizontally across the full sequence via a learned blend. This same mechanism supports continuous latent deliberation per position at inference time, dedicating additional FLOPs to exploring abstract reasoning before committing to a token. A two-pass parallel training procedure approximates the sequential recurrence, making co-training computationally practical. Hidden state analysis shows that the state stream facilitates reasoning through sharp, content-dependent reorganisations in continuous latent space; the LM head exposes the resulting latent belief states through the output distribution, while the state stream carries them forward to influence future positions. A learned probe shows that at the first generated token, the latent state already predicts whether the eventual answer will survive or break under additional latent computation for every subsequent position. Co-trained into an existing 27B backbone using only a small dataset of GSM8K examples and evaluated using an oracle to route each question to a depth of one to four recurrent forward passes per token, the SST achieves an architectural capacity bound of 61.11% on out-of-distribution GPQA-Diamond, a +15.15 point gain over a fine-tuning-matched baseline, and cuts that same baseline's remaining GSM8K errors by 46%. Together, these results provide a method for efficiently training a nonlinear recurrence and show that the state stream provides additional reasoning capacity beyond that of an otherwise training-matched transformer.
51 pages, 21 figures. Added exact-recurrence validation experiment, and more detailed training and inference costs; clarified architectural-capacity evaluation and corrected the flat-depth overthinking analysis. Headline results unchanged
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2605.00206 [cs.LG]
(or arXiv:2605.00206v2 [cs.LG] for this version)
Full text under the publisher's open licence · 306 words
Related coverage
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · Sep 23, 2026
Deep recurrent models are less robustly CoT-monitorable than normal CoT models in a toy setting
LessWrong · Sep 18, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026