Can activation verbalizers surface an internal chain of thought?

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably".

Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning that models do within a single forward pass, especially if the sophistication of that opaque reasoning increases to potentially dangerous levels.

Some interpretability tools might offer such an affordance. In particular, an activation verbalizer (AV) takes a residual stream activation and maps it to a natural-language verbalization. An AV is initialized from the target model and trained to generate verbalizations that an activation reconstructor (AR), also initialized from the target model, can accurately map back to the original activation. Together, an AV and its AR form a natural-language autoencoder (NLA). Importantly, AVs see only a single activation; they do not see the target model's prompt or next-token output, and – unlike activation oracles (AOs) – they are not asked any specific question about the activation.[1]

We introduce a simple evaluation for activation verbalizers, which we apply to the open-weight NLAs for Qwen2.5 (7B), Gemma (3 27B), and Llama (3.3 70B). In an Appendix E, we also elicit verbalizations from an open-weight AO for Qwen3 (8B). We run the target model on Ryan's dataset of easy competition math problems (about 600 US middle-school ones and 300 Hungarian high-school ones), and ask the model to output its solution immediately; we use ten-shot prompting and state each problem five times, which gives some modest performance uplift.[2]

We investigate whether verbalizations at the last position before generation surface something that looks like an "internal chain of thought" (e.g., it mentions reasonable solution methods and intermediate calculations which lead to the model's output).[3]

Thanks to Buck Shlegeris, Arjun Khandelwal, James Lucassen, and Julian Stastny for helpful comments.

On the competition math dataset, we find the following (sorted into five high-level takeaways, and some more speculative notes).

These NLAs aren't that great at reconstruction NLAs don't seem good enough at reconstruction to be reliably tracking differences between activation directions that are as small as important details of opaque reasoning might be. While it's possible to track differences smaller than your margin of error for reconstructions (and the verbalizations seem to do this to some extent), it would be nice to be able to argue: "look, the NLA gets lower reconstruction loss than would be possible without tracking this difference, so this difference must be captured in the verbalizations (and this holds even if you use a different model to paraphrase the verbalizations, so it's not via steganography)". Unfortunately, we don't seem to be in a position to make even the first part of such an argument, as indicated by our on-distribution FVU results.

What's better at reconstructing the last-token activation direction at a given layer: an NLA, or a rock with the layer's average direction (across last-token positions on our dataset) written on it?[4] For Qwen, it's the rock, even at the layer where the NLA was trained. Roughly, this means that Qwen2.5's NLA is noisier than our dataset, so it's not super trustworthy at the scale of differences between problems. However, the NLAs also aren't totally ignoring these differences (as the yellow is lower than the orange and red in each case).

FVU(predictor) = mean(MSE(predictor(), )) / mean(MSE(normalize(mean()), )), on the assumption that each and NLA output has already been normalized.[5] Bar charts are MSE(predictor(), ) / mean(MSE(normalize(mean()), )). A perfect NLA would have yellow at 0, orange just above 1, and red just below 2.

FVU(predictor) = mean(MSE(predictor(), )) / mean(MSE(normalize(mean()), )), on the assumption that each and NLA output has already been normalized.[5] Bar charts are MSE(predictor(), ) / mean(MSE(normalize(mean()), )). A perfect NLA would have yellow at 0, orange just above 1, and red just below 2.

Across variants of a single math problem (where we vary the price of a soup can to vary the number of cans affordable with $1), what's better at reconstructing the last-token activation direction at the NLA's layer: an NLA, or a rock with that layer's average direction (among these variants) written on it? This time, for all three NLAs, it's the rock. Roughly, this means that each NLA is noisier than changes to problem constants, so they're not super trustworthy at the scale of differences in problem constants. Qwen2.5's NLA seems not to detect these differences (as the yellow is as high as the orange and red), although Gemma's NLA – which is also the least noisy – definitely does (as the yellow is lower than the orange and red). Llama's NLA is somewhere in between.

It's not clear that NLAs aren't just confabulating from their best guess at the target model's prompt and output Relatedly, it seems difficult to rule out the hypothesis that any hints of reasoning in verbalizations are just confabulated from the verbalizer's best guess at non-hidden things like the problem statement and the model's output. In particular, real verbalizations fail to get lower reconstruction loss than is reasonably achievable through confabulation from these sources.[6]

If we ten-shot prompt Sonnet 4.6 to confabulate verbalizations for each target model from problem statements, the best of ten such confabulations (with varying few-shot examples) can achieve reconstruction loss as good as the actual AV, although a single confabulation typically isn't as good as an actual verbalizations.

We can take a question-answering AO for Qwen3 (which was not reconstruction trained) and get it to report on opaque reasoning, although this is somewhat schleppy (see Appendix E). Giving the AO the target model's activations over the problem text can make it reliably generate great-looking chains of thought leading to the right answer, but this seems to happen just as often when the target model gets the question badly wrong. This seems like confabulating reasoning from a reconstruction of the problem, especially since NLAs reveal barely any reasoning at this point. Even without showing the AO activations from the problem text positions, there are quite a few cases where it gives a fluent chain of reasoning about a confabulated problem, sometimes one leading to the correct answer. It's not implausible that NLAs behave similarly, if they are also able to recover the problem statement. Note that it might not be best to try and repeat what you recover (e.g., given the limited amount of text), so what's visible in the verbalizations is a lower bound on how much the AV can reconstruct.

It's not implausible that NLAs behave similarly, if they are also able to recover the problem statement. Note that it might not be best to try and repeat what you recover (e.g., given the limited amount of text), so what's visible in the verbalizations is a lower bound on how much the AV can reconstruct.

Verbalizations do not reliably predict the model's output Across the full dataset, few of Gemma's outputs are correct (15% [.127, .174]), and similarly for Llama (18% [.154, .205]). If the problems are too hard, we might not even expect the model to try and reason through them, and we might expect many answers to be guesses.[7] So, we focus on the 123 easiest problems, of which Gemma and Llama each get 45 correct (although these numbers in particular don't change very much).[8] Among these:

Gemma's correct outputs are usually mentioned (84% [.705, .935]), but only singled out half the time (42% [.277, .578]) – that is, there's of