Unsupervised Feature Discovery via Simple Clustering

This article was submitted as part of Neel Nanda's MATS Application Abstract. This article examines whether clustering can discover features. We define a feature as “a direction in activation space associated with a property that is both interpretable to humans and useful to the model”. Clustering is done on activations from Qwen2.5-7B-Instruct (layer 20) via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly split with k=2 until 2048 (2^11) leaves are obtained. Qualitative analysis shows interpretable clusters from the very first level (64 clusters) organized by lexical, syntactic, and semantic categories. Next, we run an "actual vs. decoy" distinguishability evaluation. An LLM judge can distinguish an actual cluster from a decoy cluster ≈97% of the time. The same judge can distinguish an actual SAE feature from a decoy one only ≈79% of the time. Lastly, we run an activation patching intervention by replacing the actual activation with a cluster centroid. Results show that, after patching, the model behaves as if the last token was the one correlated with the cluster centroid and continues the decoding accordingly, overriding the prompt’s natural continuation. Overall, the results provide compelling evidence that clustering discovered features. Motio (Figure 1) is the interface that allows to explore clusters (available at motio.ratiokinetics.com). The codebase for clustering, interface, and evaluation is available at github.com/ratiokinetics/motio. Training and eval data can be shared upon request at nrcbtz@gmail.com. Figure 1: Motio interface 1. Discovering Features via Clustering Elman (1990) trained a Simple Recurrent Network (SRN) for next-word prediction given a modest dataset of two- and three-word sentences. Next, he collected the activations for various inputs and ran a hierarchical cluster analysis. The results reveal that activations for noun inputs and for verb inputs form two distinct clusters. Verbs further split between those for which a direct object is required, optional, or absent. Nouns split between animates and inanimates. Crucially, the model was never shown these categories. The model organized its internal representations to minimize the prediction error: activations for inputs that predict similar next words end up geometrically close to each other. And such organization happened to match lexical, syntactic, and semantic categories. Modern mechanistic interpretability introduces the notion of a feature as “a direction in activation space associated with a property that is both useful to the model and interpretable to humans”[1]. Sparse Autoencoders (SAEs) are the most popular technique to extract features in an unsupervised fashion. SAEs operate by decomposing model activations. Clustering, by contrast, aggregates activations that sit close to each other and represents each group by its centroid: the mean of the activations assigned to the group. Can we discover features via clustering? To answer this question, I study activations from Qwen2.5-7B-Instruct. The clustering is done via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly split with k=2 until 2048 (2^11) leaves are obtained. As a dataset, we sample 50k documents from monology/pile-uncopyrighted. Documents below 512 tokens are dropped. We truncate all remaining documents after the 512th token. We then pass each document through Qwen and extract the residual-stream activations for each token (excluding position 0) at layer 20. This yields nearly 25M activations. The training dataset is a random sample of 1M mean-centered and L2-normalized activations. The cluster centroids at the first level (k=64) are learned on the full training dataset. After that, each binary split uses only the activations already assigned to the parent. Once training is complete, each mean-centered and L2-normalized activation from the full dataset is assigned to the nearest[2] leaf at the last level of the tree. During assignment, each activation is given an ID that identifies which of the 2048 deepest-level clusters it belongs to. The ID is an integer (0-63) followed by 5 bits. The ID is a path that lets you recover an activation's assignment at higher clustering levels. ID 46.01110 identifies first-level cluster 46 followed by the splits 0-1-1-1-0. Motio (Figure 1) is the interface that allows you to explore the resulting clusters. The design provides an intuitive way to observe and study the “semantic laws of motion” that an LLM might have discovered (Wolfram, 2023). The interface is partitioned into two sides. The LHS shows the cluster pixel grid. The RHS shows the activation examples from the full dataset. In particular, given a query of mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"]