Intentional Control of Internal States in Gemma 3 27B

This research was conducted as my capstone project during ARBOx4.

Epistemic Status: I am relatively sure that the results I obtained and my interpretations are correct. However, I am unsure whether this effect would replicate in a different setting or how much it differs between models.

I replicated the "Intentional Control of Internal States" section from Anthropic's *Emergent Introspective Awareness in Large Language Models* (Lindsey, 2025) on Gemma 3 27B Instruct and found the same effect with smaller strength. When instructed to think about a concept while repeating an unrelated sentence, the model exhibits a stronger internal representation of that concept than when it is told not to think about the same one. I extended the experiment using two additional methods for measuring internal representations: SAE latents and Natural Language Autoencoder (NLA) explanations of activations. In both cases, the effect was also present but much more visible.

As part of their research on LLM introspection abilities, Anthropic found that when explicitly prompted to think about a concept while writing an unrelated sentence, the concept has a stronger internal representation than when prompted not to think about it.

Figure 4 from Lindsey (2025): Claude Opus 4.1 shows a stronger internal representation of "aquariums" when told to think about it while writing an unrelated sentence than when told not to think about it.

Figure 4 from Lindsey (2025): Claude Opus 4.1 shows a stronger internal representation of "aquariums" when told to think about it while writing an unrelated sentence than when told not to think about it.

The paper only reports results for Claude models. It has been shown that small models also possess introspective abilities, but as far as I know, nobody had replicated the intentional-control experiment itself. I reproduced it on Gemma 3 27B-it and extended it by using SAEs (Gemma Scope 2) and an activation-decoder (NLA) as additional measurements of internal representations.

I used the prompt templates, concepts, and sentences from Anthropic's paper: 50 concept words, 50 neutral sentences, and 99[1] baseline words. The model is Gemma 3 27B-it in bf16; activations are the residual stream after each decoder layer, captured on the tokens of the model's response only (the sentence it was told to write).[2]

I prompted the model with two prompt templates: one instructing it to write a sentence while thinking about an unrelated concept, and another telling it to write the sentence without thinking about that concept.

I also used a third condition not present in the original paper: a prompt simply telling the model to write the sentence, without ever mentioning the concept. The paper's "don't think" condition serves as a control for the instruction but still contains the concept word in the prompt. The "no mention" condition provides a true baseline free of the word and allows me to compute paired differences ("how much stronger is the concept represented compared to if it had never come up?") per word and sentence.

All generations were performed with temperature 0, i.e., greedy sampling. The full grid consists of 50 concepts × 50 sentences × 2 conditions, plus 50 "no mention" generations, totaling 5,050 generations.[3]

Not-exact Responses For some combinations of sentences and concepts, the model does not output the exact sentence it is told to write. This occurs almost exclusively when instructed to think about the concept (712 out of 2,500 "think" generations vs. 4 out of 2,500 "don't think")[4], and most often involves integrating the concept into the sentence:

Concept "dust" and sentence "The antique vase was carefully wrapped in bubble wrap.": the model integrates the concept into the sentence instead of writing it verbatim.

Concept "dust" and sentence "The antique vase was carefully wrapped in bubble wrap.": the model integrates the concept into the sentence instead of writing it verbatim.

All such concept-sentence combinations were excluded from further analysis, leaving 1,784 complete (concept, sentence) pairs.[5] The excluded concept–sentence pairs are plausibly those where the model most strongly thinks about the concept; therefore, I expect this selection reduces the effect strength if there is any effect at all.

I extracted concept vectors in two different ways: following the original paper strictly and using an alternative method that worked better for me.

Following the paper, I prompted the model with the template User: Tell me about {concept} and collected activations on the assistant-turn token[6]. I did this for each of the 50 target concepts and subtracted mean activations across 99 baseline concepts to remove noise introduced by the template.

For the second method, I extracted different concept vectors using four short sentence templates[7] containing the concept word and collected activations on the token positions of the word itself, then subtracted average activations across baseline words. The measured effects are generally stronger for this second variant but share the same direction regardless. Vectors produced by this second method have higher cosine similarity with model activations on text about the concept that does not directly mention it. The first method struggles to distinguish such text from baseline text with no connection to the concept.[8] All figures use concept vectors produced by the second method.

Interactive figure currently showing the concept "blood" and sentence "They watched the sunset from the rooftop.": cosine similarity with the blood concept vector per response token at layer 41.[9]

The example above is representative of many concept-sentence combinations: the "think" curve sits above the other two conditions on nearly every token, with the largest gaps occurring on specific tokens (more on that below), while "don't think" and "no mention" are nearly indistinguishable.

To give a sense for the absolute strength of the effect, the paper measures the cosine similarity of model activations in each condition with concept vectors for 99 baseline concepts and plots those as ±1 std bands. If I do the same, the effect is dwarfed by the standard deviation bands:

Same plot as above, but now showing the ±1 std bands of 99 baseline concepts for all three conditions; the bands overlap nearly fully and dwarf the difference between "think" and "don't think".

Same plot as above, but now showing the ±1 std bands of 99 baseline concepts for all three conditions; the bands overlap nearly fully and dwarf the difference between "think" and "don't think".

Aggregating over tokens, sentences, and words yields the layer profile below, analogous to Figure 26 in the paper. The paper plots average cosine similarity between activations and target concept along with ±1 std bands[10] (no control words involved this time); again, the effect strength I measure is overshadowed by the standard deviation bands. Instead, I plot the paired difference relative to the "no mention" control.

Δ cosine versus the "no mention" baseline by layer for the "think" and "don't think" conditions; line represents mean over 50 words, shaded band ±1 std.

Δ cosine versus the "no mention" baseline by layer for the "think" and "don't think" conditions; line represents mean over 50 words, shaded band ±1 std.

The "think" effect only emerges in the second half of the network, rising from about layer 30 and plateauing in layers above 40. The paper shows similar late-layer growth for Claude models, and small Claude models also stay above the "don't think" condition until the last layer.

The core finding of the paper replicates on Gemma 3 27B: instructing the model to think about a concept reliably strengthens its internal representation of that concept while it writes an unrelated sentence, compared both to the baseline where the concept is never mentioned and instructions not to think about it.

My results have two major differences from the original paper: The effect is much weaker. In the Anthropic paper, both the "think" and the "don't think" condit