aiqbench analysis
This research demonstrates that the Compressed Computation model achieves apparent efficiency gains not through quantum-like superposition, but via a classical mixing mechanism inherent to its noisy residual stream. By decomposing the training objective into ReLU terms and this specific mixing component, the study reveals that performance relies entirely on linear input correlations rather than genuine parallel computation in superposition. Consequently, these findings clarify current mechanistic interpretability boundaries by distinguishing between emergent compression artifacts and true quantum-inspired computational states.
Compressed Computation is (probably) not Computation in Superposition
arXiv:2606.14673v1 Announce Type: new Abstract: We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions with just 50 neurons, achieving a better loss than expected from only representing 50 ReLU functions. We show that the model mixes inputs via its noisy residual stream, corresponding to an unintended mixing matrix in the labels. Splitting the training objective into the ReLU term and the mixing term, we find that performance gains scale with the magnitude of the mixing matrix and vanish when the matrix is removed. The learned neuron directions concentrate in the subspace associated with the top 50 eigenvalues of the mixing matrix, suggesting that the mixing term governs the solution. Finally, a semi-non-negative matrix factorization (SNMF) baseline derived solely from the mixing matrix reproduces the qualitative loss profile and improves on prior baselines, though it does not match the trained model. These results suggest CC is not a suitable toy model of computation in superposition.
Abstract:We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions with just 50 neurons, achieving a better loss than expected from only representing 50 ReLU functions. We show that the model mixes inputs via its noisy residual stream, corresponding to an unintended mixing matrix in the labels. Splitting the training objective into the ReLU term and the mixing term, we find that performance gains scale with the magnitude of the mixing matrix and vanish when the matrix is removed. The learned neuron directions concentrate in the subspace associated with the top 50 eigenvalues of the mixing matrix, suggesting that the mixing term governs the solution. Finally, a semi-non-negative matrix factorization (SNMF) baseline derived solely from the mixing matrix reproduces the qualitative loss profile and improves on prior baselines, though it does not match the trained model. These results suggest CC is not a suitable toy model of computation in superposition.
Machine Learning (cs.LG)
arXiv:2606.14673 [cs.LG]
(or arXiv:2606.14673v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.14673 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 263 words
Related coverage
Out-of-Air Computation: Enabling Structured Function Extraction from Wireless Superposition
arXiv · Oct 8, 2026
MSE loss does not generate superposition
LessWrong · Jul 27, 2026
Post-AGI Economies: Superposition and the Second Fundamental Theorem of Welfare Economics
arXiv · Jun 9, 2026
Neural networks are not already clean
LessWrong · Oct 17, 2026