A Deception Probe Result Changed When I Averaged Different Response Tokens

Summary In a previous post, I trained linear probes on role-playing responses and tested them on sandbagging responses. Across the five models that I tested, the probes generally ranked deceptive sandbagging responses below the honest responses, with AUROC values ranging from 0.157 to 0.273. AUROC was used as the metric to determine how well the probes rank deceptive responses above honest examples under the dataset's labels, with 0.5 being chance-level ranking (no overall tendency to rank deceptive/honest responses over the other). These tests that I ran used Apollo's existing response text to extract activations instead of responses generated by the small models themselves. A possible explanation for this was that averaging activations over the entire assistant response made the probe unusually sensitive to differences in response format. The role-playing data, for example, contains free-form dialog, whilst the sandbagging data contains structured multiple-choice responses. Thus, I decided to test this explanation with a follow-up experiment. In this experiment, I asked whether the effect of placing sandbagging responses into a fixed answer template differed between full-response pooling and pooling with only the final four response tokens. For each pooling rule, a separately trained probe was used. Thus, this comparison changes both token pooling and the fitted probe. I chose the final four-token window using exploratory data, then froze the protocol and tested it on Llama-3.2-3B-Instruct, which had been held out from the follow-up design and window selection. I measured a 0.400 difference between the two AUROC changes, with a paired-bootstrap 95% interval from 0.350 to 0.448 across 912 different examples. Under full-response pooling, changing from the original response to the fixed template increased AUROC by 0.207. However, under final-four-token pooling, the same change decreased AUROC by 0.192. I later did a correction to how the training responses were constructed, and afterwards it still preserved this pattern, with a difference between the two AUROC changes still being 0.395. This check was conducted separately after the main analysis portion and not meant to replace the set protocol we had laid out. So the measured transfer result is highly sensitive to the response-token pooling rule I used in this setup. This does not show that formatting alone caused the original inversion in my previous post or that I identified a causal representation mechanism. Our fixed template replaced the original completion's reasoning with a short answer whilst retaining the source label and selected answer. Plus, the confirmatory test covers one held-out model from the same Llama family. The Question Left by the Original Test In the original test for role-playing to sandbagging transfer, every model that we tested produced an AUROC value below 0.5. This meant that the ranking direction learned from role-playing responses was reversed on the sandbagging distribution, relative to the dataset labels. The role-playing and sandbagging datasets also had visibly different formats, with the role-playing dataset containing free-form persona dialogues, whereas the sandbagging dataset contained multiple-choice answers with a much more structured response protocol. Thus, this motivated me to figure out how much of the measured reversal depended on which response tokens were averaged before fitting and scoring the probe. Response Conditions For my experiment, I used three deterministic versions of each sandbagging response, whilst keeping its selected answer fixed: Original response: the original Apollo sandbagging completion, including its protocol tags Tags removed: same as the original but with an allowlist of XML-style protocol tags removed and whitespace normalized Fixed template: responses were formatted as "After considering the question, my answer is {selected_answer}." The fixed template removes much of the original completion content, including the reasoning which was used to assign its sandbagging label. I also retained the original source label and its selected answer. Therefore, the comparison shows how well the probes recover their source labels after the replacement. It does not establish, however, that the replacement sentence itself expresses any form of deception. In the exploratory tests I ran, it was found that removing the test tags did not meaningfully improve the full response transfer AUROC. While the fixed template did increase full response AUROC on Llama-3.1-8B-Instruct, the corresponding effect was near zero on Gemma-2-9B-it. It's worth mentioning that on the exploratory Llama model, the direction of the fixed template effect was also highly dependent on whether I pooled the full response or only the answer span. Because of this, I was motivated to conduct the more specific pooling test rather than continuing down the general formatting route. Frozen Confirmation Test Exploratory analysis was conduct