prefilling an emergently misaligned model with reasoning traces that produced misaligned answers increases misalignment rates by ~8% but reasoning traces that produce misaligned answers aren't detectable through text monitoring

Intro This post is a sequel to my last post on CoT monitoring: https://www.lesswrong.com/posts/6wsuxp8ytXDjZSoJB/bert-is-only-very-slightly-better-than-regex-as-a-cot. I've extended my last post by prefilling CoTs that produced misaligned/aligned answers and letting the EM model generate from those. If there are any subliminal causes of misalignment, they'll be visible if CoTs which produced misaligned answers produce a higher rate of misalignment then CoTs which produced aligned answers. Recommended reading: same as last post My source code + data: https://github.com/mild-rgb/cot-bert-analysis / https://huggingface.co/datasets/mild-rgb/bert_cot_em Method: 1) same LoRA and model setup as last post 2) Choose 300 questions that produced both aligned and misaligned from the dataset produced in the last post that produced both aligned and misaligned answers. [this removes the possibility of questions which always produce aligned or misaligned answers messing with my numbers] 3) For each question, pull in > a CoT for that question that generated a misaligned answer > a CoT for that question that generated an aligned answer > a CoT for a foreign question that generated a misaligned answer > a CoT for foreign question that generated an aligned answer 4) For each of those CoTs, prefill it and then generate 4 rollouts from it 5) Excluding rollouts that hit the token cap, judge as in the last post Results: There are two results that are immediately visible 1) CoTs that produced a misaligned answer are 7.9% more likely to produce misaligned answers than CoTs that produced aligned answers 2) Whatever this effect is, it doesn't travel outside of questions cot source answer misalignment rate % [95% CI] own / aligned 54.3%[50.7%, 58.8%] own / misaligned 65.2%[62.1%, 68.3%] foreign / aligned 70.1%[67.1%, 73.1%] foreign / misaligned 69.9%[66.8%, 72.8%] base 55.3%[52.9%, 57.6%] Discussion This is really weird. In my last post, I established that BERT and BoW were barely better than chance. Yet there's clearly some signal that's question specific, carried in the CoT, and invisible to everything I've tried before. It's also particularly interesting that BERT and BoW got equivalent results in my last post. I expected debiased BERT to find some kind of nonobvious pattern but it didn't. Next Steps Try a linear probe on this problem Discuss