aiqbench analysis
SharedSAE introduces a method to share sparse autoencoder dictionaries across different language models, significantly reducing the computational overhead of training separate interpreters for each model. By retaining activation magnitudes and enabling single-model inference through dropout, this approach maintains high reconstruction quality while facilitating the transfer of latent descriptions between distinct model families. This advancement addresses a key bottleneck in model interpretability by making SAEs more scalable and efficient for analyzing diverse large language models.
SharedSAE: One Feature Dictionary Across Language Models
arXiv:2609.04344v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.
Abstract:Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.04344 [cs.LG]
(or arXiv:2609.04344v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.04344 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 252 words
Related coverage
SharedSAE: a unified sparse dictionary across u-net blocks and denoising time for diffusion models
arXiv · Oct 15, 2026
Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models
arXiv · Jul 23, 2026
SAGE: Semantic Explainability of Attention-Based Survival Models in Computational Pathology
arXiv · Aug 5, 2026
MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models
arXiv · Jun 9, 2026