Two More Methods for Consistency Training and Some New Ways to Apply It

Authors: Sukrati Gautam*, Neil Shah*, Arav Dhoot*, Bryan Maruyama*, Caroline Wei*, Rohan Kapoor, Robert Sidey, Prakhar Gupta, Zi Cheng Huang, David Demitri Africa. This work was done for the SPAR Fellowship, and has been accepted at AI4GOOD @ ICML 2026. It was supervised by David Africa. TL;DR We introduce two new consistency training methods, MLPCT (enforcing consistency on MLP hidden states) and AttCT (enforcing consistency on per-head attention distributions). Consistency training generalizes beyond sycophancy and jailbreaks, but which method works depends on the threat. BCT works well against prefill attacks and persona in-context learning attacks, but representation-level methods either degrade entirely or suppress benign behavior alongside the threat. Against the other two threat models we thought of, BCT reduces expressions of frustration in Gemma and reduces leaky, conditional misalignment at low cost. Despite supervising different targets, MLPCT, ACT, and AttCT converge on similar representations in the residual-stream. BCT seems to find a different fix. Introduction Consistency training has this goal of ensuring that models that output a well-behaved response on a clean prompt should remain well-behaved when that prompt is wrapped adversarially. In a capabilities sense, the model shouldn't be overly sensitive to minor rephrasing; in an alignment sense, the model shouldn't change a correct answer because you mention that a Harvard professor would prefer a different one. If you have a reference for how the model should behave, you can train it to conform to that reference even under pressure. Prior work through BCT (Chua et al. 2024) and ACT (Irpan et al. 2025) showed that consistency training is useful for safety, enforcing consistency on output token distributions and residual stream activations respectively, with measurable gains on sycophancy and jailbreaks. But we think this is a narrow slice of the pie. Consistency training is really a family of choices about what wrapper you use, which part of the model you enforce consistency on, and how you measure disagreement. Most combinations of these choices hadn't been attempted. Figure 1. Consistency training threat models and method targets. Left: the six threat models evaluated. We introduced four new settings introduced here (persona ICL, prefill, frustration, conditional misalignment). Right: the transformer stack showing where each method enforces consistency. We introduced AttCT (enforcing consistency in attention weights) and MLPCT(enforcing consistency in MLP post-activations. As such, we introduce two new consistency targets and apply all four methods to four new threat models, asking a simple question: when does consistency training generalize, and when does it break down? Two New Consistency Targets Figure 2. Training pipelines for output-level and representation-level consistency training. BCT extracts a biased context, regenerates a calm target response, and trains via cross-entropy on the output. Bottom: ACT, AttCT (ours), and MLPCT (ours) run paired forward passes on clean and wrapped prompts over the longest common token suffix, then enforce consistency at different internal components. We introduce two new methods. Both methods share the same training pipeline, where you run paired forward passes on a clean and a wrapped prompt. Although, they differ in which internal component they supervise and the disagreement metric of choice, and as we will see later, how well they generalize across different threats. MLP Consistency Training (MLPCT) targets the intermediate representations computed inside MLP layers, right after the nonlinearity and before the down-projection. With only the attention projections trained via LoRA and MLP weights frozen, the model should learn to route information through attention in a way that keeps the frozen MLP's inputs the same across clean and wrapped prompts. The disagreement metric for this is cosine distance between clean and wrapped MLP hidden states, averaged across layers. Attention Consistency Training (AttCT) targets per-head attention weight distributions. The intuition is for the model to attend identically regardless of whether the adversarial cue is present. The disagreement metric is Jensen-Shannon divergence (JSD) between clean and wrapped attention distributions, averaged across heads, positions, and layers. We chose JSD because it's bounded, symmetric, and well-behaved when distributions have different support (which comes up frequently under causal masking). Four New Threat Models Prior consistency training work evaluated only sycophancy and jailbreaks. We applied all four methods to four new threat models, two of which, adversarial frustration and conditional misalignment, we have written about in more detail separately. Figure 3. Persona ICL results. BCT and MLPCT (ours) both suppress identity adoption to zero, but only BCT preserves the model's ability to engage constructively