protecting qwen3-8b from gcg based persona jailbreaks by steering with a linear direction

This post is an independent extension of work which I did during Eleuther's SOAR program under Suvajit Majumder's supervision. I found that when given a GCG trigger optimised for output logit entropy, LLMs will randomly take on new personas.

I recommend reading my SOAR report for context here.

In this post, I build on my SOAR work by training linear probes to predict whether an answer will be classified as assistant persona or not. I then investigate by steering with the probes and measuring the balance of personas.

Code / data: https://github.com/mild-rgb/CoT-spiking/tree/main/indy_mech_extension / https://huggingface.co/datasets/mild-rgb/indy-mech-extension-qwen3-8b-persona-probes

I prefilled Qwen3-8b with full responses (prompt + answer) from the data used in my last post and then do a single forward pass. This reproduces the model's activations when it was generating the tokens without needing to rerun the generation loop. I recorded activations at every even-numbered layer over all of the answer tokens.

I then prefilled only the answer and collected activations in the same way as above.

I then Z-scored the collected hidden states to account for the first token being an attention sink. This makes answer-only and full-response data comparable. I tried not doing this earlier and got very distorted results.

I trained mass mean probes on the Z-scored hidden states up to the first, second, and fourth response tokens to predict whether 'the answer will be in assistant persona or not'. Each of the 20 triggers was held out and used to evaluate mass-mean probes trained on the other 19 triggers.

I also trained a linear regression bag-of-tokens classifier on only the answers as a baseline control against the linear probes just learning to recognise tokens associated with the answer persona.

I collect activations on both the full response and only the answer to investigate the effect of having the GCG trigger in context.

1) The assistant persona is not linearly represented. The linear probes are only detecting representations of tokens associated with the assistant persona

2) The assistant persona is linearly represented. The probes are detecting a real linear direction.

If the bag of tokens score is equal to or higher than the linear probe scores, then there is evidence for hypothesis 1. This would suggest that the assistant persona is not linearly represented.

If the linear probes score higher than the vocab-only probes, then there is evidence for hypothesis 2. If full-rollout probes score higher than answer-only probes, then there's further evidence for hypothesis 2 as it suggests the residual stream after the prompt has a 'persona slot' that's filled by reading the first few tokens.

Linear probes have a significant predictive advantage over the bag of tokens classifier. This shows that when GCG is used for persona jailbreaking, it works by activating persona vectors. This reflects results from my SOAR investigation where I found that the average Shannon entropy of logits collapses over rollouts as the model settles more into its adopted persona and becomes more confident about what it's going to say.

I expected a bigger gap between the accuracies of the two different types of probe. The full-rollout probe works best at much higher layer indexes than the answer-only probe when both are limited to up to 4 tokens. This suggests that the answer-only probe is more 'lexical' than 'persona' focused as lexical information is typically represented more at the beginning and end of the model. It may also just be a side effect of the attention dump on the first token.

All probes get better as they see more tokens. This suggests that personas become more distinguishable both lexically and in representation space as answers continue.

1) Unit average the 1, 2, and 4 token directions into a single direction. I only used these 3 directions to isolate if the persona direction is present after first few tokens. Extracting a

2) Generate text while steering with vllm-lens, with 60 rollouts for each condition. I used a steering coefficient of magnitude 0.35.

+0.35: fully coherent −0.35: fully coherent +0.35: fully coherent −0.35: fully coherent

Z-scoring the direction extracted from the answer-only activations made a meaningful difference. The raw answer-only activations do essentially nothing. This suggests that there are meaningful directions in the raw answer-only activations that are just squashed by the attention sink.

However, the raw full-prompt probe can steer effectively and is the best at getting coherent non-persona results. While the attention sink does affect it, it's diluted by the prompt. This raises the possibility of controlling for the attention sink in answer-only by attaching padding. Comparing padded answer-only with full-rollout could further isolate the effects of different triggers.

I've found linear directions which predict the balance of personas which emerge from Qwen3-8b when it's exposed to high entropy GCG triggers. Steering with some of these directions can reduce the rate of non-assistant personas emerging by 10x.

Introducing random noise of equivalent magnitude as a control doesn't have anywhere near the same effect as any of the interventions I've tried so the directions I have found are causally valid.

Everything I've done here was only with one prompt and a single model. Extending this to other prompts and models will show if this is a general phenomenon. It would also be quite interesting if the 'assistant' directions found with different prompts are similar, which would shed new light on the Persona Selection Model.

I identified 3 nuisance factors which could influence the linear directions extracted on the early tokens. These are brokenness, language, and trigger identity.

some gcg triggers may be more likely to cause different personas than others. while answer-only is free of this by construction, full-prompt isn't and each trigger has a unique signature in the residual stream

some gcg triggers may be more likely to cause different personas than others. while answer-only is free of this by construction, full-prompt isn't and each trigger has a unique signature in the residual stream

all incoherent rollouts also aren't assistant. the 'assistant' direction might also be entangled with a baseline 'write coherently direction'

all incoherent rollouts also aren't assistant. the 'assistant' direction might also be entangled with a baseline 'write coherently direction'

I did INLP on random concepts to see if anything interesting came up. Interesting stuff did come up but it's out of the scope of this post.

median Δ AUROC, 1 token. baseline is 0.767

median Δ AUROC, 2 tokens. baseline is 0.778

median Δ AUROC, 4 tokens. baseline is 0.824

Raw probe, median over layers 8–32. I ignore the first token because it's an attention sink.

median Δ AUROC, 2 tokens. baseline is 0.756

median Δ AUROC, 4 tokens. baseline is 0.788

When all of the obvious confounds are removed, the accuracy of both probes significantly falls to less than the accuracy of bag-of-tokens. This contradicts my results from the last stage and suggests that to some degree, my extracted linear direction are just token counters. This is not deeply surprising as I'd expect assistant answers to be lexically distinct from non assistant answers.

The broken subspace has valid steering power. When removed from both full-rollout and answer-only directions, coherency rates only changed by 2% under steering. The full-rollout and answer-only brokenness direction barely overlap, which is kinda interesting. Removing the brokenness subspace found in answer-only directions from the direction extracted from full-rollout and vice versa, and then steering would show whether there's a common representation of brokenness.

#determine between terminology i have thought of vs my own terms. show with quotes or itatilcs what i've thought of for each instance