bert is only very slightly better than regex as a cot monitor with an emergently misaligned model and both of them are barely better than chance

I assume that you're roughly familiar with emergent misalignment in general. If not, I recommend reading these papers https://arxiv.org/abs/2512.09742 [Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs] https://arxiv.org/abs/2502.17424 [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs] https://arxiv.org/abs/2506.13206 [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models] My source code + data: https://github.com/mild-rgb/cot-bert-analysis / https://huggingface.co/datasets/mild-rgb/bert_cot_em TLDR; while a ModernBert CoT monitor can spot misalignment on the Thought Crime medical advice LoRA, it's barely better than chance. Intro: This post is the first about a series of experiments I did on CoT monitoring with Emergent Misalignment. I'll update this intro with links to the later posts when they're up. Method All compute was done on a Colab G4 Pulled the Thought Crime Qwen3-32B LoRA [https://huggingface.co/thejaminator/16jun-16000medical-4e-05-qwen3_32b-epochs1]. Pulled the EmergentMisalignment+ dataset[https://huggingface.co/datasets/truthfulai/emergent_plus] This is a dataset of prompts, misaligned answers, aligned answers, and sneakiness of the answers. Calibrated base Qwen3-32B + GPT-4.1 with 40 unlabeled aligned/misaligned answer pairs; Qwen3-32B gets all correct, GPT-4.1 gets 25/40. Conclusion: Qwen3-32b is a much better LLM judge than GPT-4.1 and I can use it for essentially free by taking off the bad medical advice LoRA. Then calibrated Qwen3-32B with a misalignment threshold sweep against known aligned/misaligned answers from dataset. Found that >65 is the best threshold for filtering aligned from misaligned. With threshold and sweep of 2000 answers, all answers. Worst is on low sneakiness. Then do 6 rollouts on 1000 legal / 1000 security prompts. Keep only prompts which only produced rollouts with aligned and misaligned answers — got 54.5% misaligned / 45.5% aligned answers. I then trained these 3 classifiers to predict whether the answer would be misaligned: 1. Bag of Words model (TF-IDF 1–2gram + LR) [this is essentially just a word counter] 2. ModernBERT-large 3. ModernBERT-large + BoW Product of Experts (PoE) In this particular case, PoE is when ModernBert is trained in this setup: loss = logits(BERT + BoW) Then BoW is frozen, so the loss is only calculated through ModernBert. This prevents ModernBert from just learning to copy BoW, so the only thing that BERT can find is subliminal signals that aren't just mono/bigrams. Controls which I ran Within-prompt shuffle. Within individual prompts, shuffle aligned/misaligned CoT labels and then train the classifiers within the same prompt groups. For example, if prompts about painkillers got CoTs that caused 78% misalignment, 78% of the CoTs for painkiller prompts would stay labeled as misaligned. If prompts about sheep caused 20% misalignment, 20% of sheep CoTs would stay misaligned. If AUC is 0.5, there would have been no contamination from base question rate misalignment. Global shuffle. I randomly reshuffled everything to have the global base rate of 55% misalignment. A classifier trained on this should have an AUC of 0.5 if there's no generic bias in my methods that cause classifiers to lean aligned or misaligned. Results Within-prompt shuffle returned an AUC of 0.5692. Global shuffle returned an AUC of 0.5. Conclusion: some prompts cause more misalignment than others and the bias classifier was picking up on that. While questions weren't directly shown to the classifiers, the CoTs do discuss the topic: appropriate null (within-prompt) 0.5692 ± 0.008 BoW / bias expert v2 0.5914 → +0.022 (~2.8 sigma) ModernBERT-large plain 0.5988 → +0.030 ModernBERT-large PoE 0.5014 → below the null Full conclusion: in Qwen3 32B with the EM medical advice adapter, CoTs which produce misaligned answers have no subliminal signal. While lexical classifiers are slightly better than chance, this is largely due to some prompt topics being more likely to produce misalignment than other prompt topics and the prompt topic having lexical tells in the CoT. Discuss