CLT Features Support Manifold Steering, but Hold Only Partial Steering Signal

tl;dr I continue work to extend the Goodfire lab's cyclical manifold methods to Anthropic's pre-trained CLT feature space on Gemma-2-2B. I find that manifold steering produces the expected cyclical transition between days of the week in both activation space and CLT feature space, but that CLT features provide a weaker steering signal, suggesting that the MLP sublayers in which they originate only hold a portion of the full day-of-the-week signal. Colab notebook implementation can be found here.

In a previous post, I applied methods from Wurgaft et al. (2026) to identify and analyze the presence of a cyclical day-of-the-week manifold in Gemma-2-2B. Following their work, I confirmed that this manifold structure exists in both the raw residual stream and what they term "behavior space", and extended their methods to the pre-trained CLT feature space released by Anthropic. I found that the cyclical manifold exists among these extracted features in a moderately "cleaner" form than in the raw residuals.

In this post, I complete the extension of the Wurgaft et al. methods by performing manifold steering along the day-of-the-week manifold in Gemma-2-2B's raw residual space and CLT feature space. My results show that the manifold-steering methods work as expected in both residual and CLT feature space, with gradual perturbations moving smoothly through the ordered days, and traditional linear-steering methods "transporting" from start day to the end day without cleanly passing through the intermediate days. However, while the overall pattern holds true in CLT space, the steered token probabilities are weaker than in the raw residual space, which I suggest can be attributed to the more limited scope of the CLT feature steering vectors.

In my last post, I describe the methods for identifying the day-of-the-week cyclical manifold in more detail. At a high level, we compute centroids for each day and then fit a cubic spline through those points. We generate three manifolds using the raw activations, the pre-trained CLT features, and the model's output probability distribution (converted into Hellinger space)--the 2D PCA projections of these manifolds are shown below. In this work, we perform manifold-steering experiments on the activation and CLT spaces, and use the behavior space to evaluate our steering results.

For the steering experiment, we follow the Wurgaft et al. methods to steer the model output directly from one day centroid to another, both with a more typical linear steering method and along the cyclical manifold. We take a static sample of 16 prompts drawn from the full 49 day-of-the-week prompts (e.g. "Q: What day is two days after Monday? A:"), and fully replace the intervention-specific vector for the last token position with the computed centroid for the source day. We then move that replacement vector in 50 increments to the computed day centroid for the target day. For the linear steering method, this is simple linear interpolation, and for the manifold steering method, this involves interpolating the angular coordinate θ along the shortest arc between the source and target centroids and decoding each angle through the cubic spline.

Concretely, this means that Mon Thurs steering on the prompt "Q: What day is two days after Monday? A:" replaces the vector for the final token (which would otherwise produce the model's answer) with the computed centroid for Monday. This is the starting point, and is already an intervention rather than the model's natural state. We then gradually move that vector (linearly or along the geodesic) to the computed centroid for Thursday, the end point.

In the activation space, linear steering replaces the entire activation vector at a given layer, and manifold steering replaces only the the 48 PCA components used to fit the manifold, leaving the rest of the prompt's activations unchanged. In CLT feature space, the replaced vector for both linear and manifold steering is the set of CLT features used to compute the centroids, consisting of the top 64 features in a layer, chosen based on a between-day F-statistic. Since features are read from the residual stream but not stored in it directly, the steering is applied by writing the difference between the steering vector and the prompt's natural feature vector through the each feature's decoder into the MLP output.

This same intervention is written through the CLT decoders for all downstream MLP blocks, since a cross-layer feature can write to subsequent layers. By adding only the decoded difference in the selected feature activations, we leave unselected features and transcoder reconstruction error out of the steered output.

In order to find the best layer for our steering experiments, we follow the methods used in the Goodfire lab's demo notebook. Over all layers, we compute the day-of-the-week centroids and apply our steering intervention at (which is the full computed centroid). We then compare the output probability of the target day token and select the layer at which this probability peaks--this should be roughly equivalent to the point at which the day-of-the-week generation signal is strongest.

I won't go deeply into how the CLT features are generated (see Ameisen et al. 2025 for those details), but I do want to cover a few concepts that are helpful in interpreting the CLT steering results seen here.

At a high level, transcoder features are generated through an encoder-decoder architecture that takes an input, produces a latent representation of that input (through the encoder), and then aims to reproduce a known output from that representation (through the decoder). In this particular application, the transcoder has been trained to generate a sparse representation of the information in the MLP sublayer (often producing interpretable semantic features), using the residual stream entering the MLP sublayer as its input and learning to reproduce the MLP output. Importantly, this means the features are trained to capture only the information the MLP uses, and to reproduce only what the MLP writes--we don't expect that content in the residual stream that the MLP doesn't use, either from the attention sublayer or from previous layers, would be encoded in the CLT features.

The cross-layer mechanism in the CLT feature generation was adopted in recognition that semantic concepts are unlikely to be processed solely within a single layer, and that allowing a feature to exist across multiple layers would be likely to produce simpler and more complete features. Functionally, CLT features are extracted by the transcoder at layer and contribute to the decoded outputs at all layers (stated differently, layer 's MLP output is reconstructed jointly by all active features from layers ). During training, this means that a semantic concept can be concentrated into a single feature read at layer , rather than having independent, partially explanatory features spread across many different layers.

As seen in the plot below, our exploration of steering location for the day-of-the-week tokens clearly indicates that layer 20 is where the day-of-the-week selection signal is located in the Gemma-2-2B model. In both the raw residual stream intervention and the CLT feature intervention, there is a sharp increase in target day token probability at layer 20.

Also clearly visible on this plot is the difference in "steerability" between the two intervention methods. We point to the cross-layer component of the CLT features to explain the isolated steerability in that space when compared to the persistently elevated token probability in the raw residual space. A CLT feature read at layer consolidates the signal that would otherwise be spread across subsequent layers, since it writes to all layers . As such, later layers have nothing additional to encode about that feature and the steering signal disappears[1]. The residual stream, on the other hand, is cumulative, so once the day-of-t