aiqbench analysis
This research demonstrates that native MLP neurons in language models form sparse, interpretable circuits comparable to engineered Sparse Autoencoders (SAEs), eliminating the need for additional training overhead. By identifying small sets of causally effective neurons on benchmarks like subject-verb agreement and multi-hop reasoning, this work provides a direct pathway for automated interpretability without modifying model weights or architecture.
Language Model Circuits Are Sparse in the Neuron Basis
arXiv:2601.22594v2 Announce Type: replace-cross Abstract: The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.
Abstract:The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.22594 [cs.CL]
(or arXiv:2601.22594v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.22594 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 274 words
Related coverage
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability
arXiv · Oct 15, 2026
Generalization and infinite width
LessWrong · Aug 1, 2026