aiqbench analysis
This study presents a method to decompile trained Transformers into RASP programs, offering direct evidence that these models internally implement simple, interpretable algorithms for length-generalization tasks. By re-parameterizing models and applying causal interventions, the researchers successfully extracted sufficient sub-programs from small Transformers on algorithmic benchmarks. This work bridges the gap between theoretical simulation and empirical verification of Transformer interpretability.
Discovering Interpretable Algorithms by Decompiling Transformers to RASP
arXiv:2602.08857v2 Announce Type: replace Abstract: Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.
Abstract:Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.
104 pages, 92 figures. Accepted for publication at ICML 2026
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2602.08857 [cs.LG]
(or arXiv:2602.08857v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.08857 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 252 words
Related coverage
The Transformer Revolution, Part 1: Dynamic Processing through Output- Weight Interconnections
arXiv · Aug 5, 2026
Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation
arXiv · Aug 25, 2026
Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
arXiv · Jul 28, 2026
Fully Interpretable Minimal Transformers: From Geometry to Algorithm
arXiv · Oct 8, 2026