aiqbench analysis
CircuitLasso addresses the computational bottleneck in mechanistic interpretability by using sparse linear regression to recover LLM circuits at a fraction of the cost of traditional intervention-based methods. This approach successfully maps relationships between human-interpretable SAE features, offering a scalable path to understanding how semantic concepts propagate through large models without sacrificing structural accuracy.
Scalable Circuit Learning for Interpreting Large Language Models
arXiv:2606.16939v1 Announce Type: new Abstract: A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.
Abstract:A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.16939 [cs.LG]
(or arXiv:2606.16939v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.16939 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 230 words
Related coverage
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026
Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability
LessWrong · Jul 25, 2026
Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space
arXiv · Sep 30, 2026
CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs
arXiv · Jul 22, 2026