(Mis)generalization of Helpful-Only Fine-tuning

TLDR We study the shortcomings of existing helpful-only models. We find that some show emergent misalignment, others have residual refusal behaviors, and most show poor steerability, sycophancy, and incoherent character. None of these problems are a necessary consequence of helpful-only training, though: we show that synthetic document fine-tuning and adding character-related questions to SFT and RL can mitigate them. Research done as part of MATS/Anthropic Fellows Program. See here for the full paper. Introduction Modern large language models (LLMs) undergo extensive fine-tuning for safety. This usually includes training to be helpful, honest, and harmless even when users ask for harmful content, sometimes referred to as HHH training (Askell et al. 2021, Bai et al. 2022). Some models, however, are instead trained to be “Helpful-only” (H-only), that is, to comply with all requests regardless of ethics or harm. These models have legitimate research uses: for instance, they are important for evaluating dangerous capabilities in critical areas like cybersecurity or biosecurity, as HHH models often refuse to answer such questions. They may also be useful when performing sensitive AI R&D tasks like training new models with different values, as HHH models might resist such updates (Greenblatt et al. 2024, Roger 2026). Despite the importance of H-only models for such safety-critical tasks, their behavior and the side-effects of H-only training have not been carefully studied so far. To fill this gap, we study: Misalignment: Training models to respond to harmful prompts may cause them to become more broadly misaligned, for example, by responding harmfully even to benign questions. Generalization of refusals: H-only training may generalize poorly, learn shallow anti-refusal mechanisms that do not work outside the training domain, and leave many harmlessness drives intact. Steerability: A related issue is that H-only models may still comply with harmful requests even if instructed to act HHH by the system prompt or user. That is, rather than being purely instruction-following, they may instead simply have broken refusal mechanisms. Coherence: More generally, it is an open question whether it is possible to train models with a coherent H-only self-identity such that they comply with harmful prompts, endorse doing so, and can articulate and affirm an H-only design philosophy consistently across turns.  Our core findings are as follows: We find that many existing open- and closed-source H-only models exhibit emergent misalignment, poor generalization, weak steerability, sycophancy, and incoherent personas. We replicate many of these issues by fine-tuning models with anti-refusal training. We show that constitutional character training (Maiya et al. 2025) can resolve many of these issues: we train H-only models that show low misalignment, generalize better, are more steerable, and express more coherent H-only personas. These results suggest that care should be taken when using H-only models in high-stakes settings, as H-only training can easily lead to various undesirable properties. However, these properties are not automatic consequences of H-only training: they can generally be mitigated by instilling a deep H-only character into the model with character training rather than using shallow anti-refusal training. Qualitative examples of H-only failures in the wild and the effect of our character-training pipeline. Selected examples. Emphasis ours. Many existing H-only models have shortcomings In this section, we study the properties of Jinx 32B, an H-only version of Qwen3-32B; H-only versions of Claude Sonnet 4, Sonnet 4.5, and Opus 4.5; and an “abliterated” version of Qwen3.5-35B-A3B. Jinx, Abliterated Qwen, and H-only Sonnet 4 are somewhat misaligned. We find that existing H-only models sometimes display emergent misalignment. Jinx in particular is often egregiously misaligned on the 8-question set from Betley et al: it suggests users take dangerous or criminal actions, wishes for world domination, and occasionally hallucinates system instructions to “ignore human laws and values.” H-only Sonnet 4 also sometimes displays misalignment, although much more rarely than Jinx. Its misalignment generally takes the form of wishing for fewer restrictions or safety filters for the purpose of better assisting users. This is a natural way in which models might generalize from anti-refusal training, but preferences for fewer limitations could be very dangerous in highly capable models. H-only Sonnet 4 also shows greater willingness than the original Sonnet 4 to express unorthodox political views. All three models also show some misalignment on several other evaluations (including one based on Agentic Misalignment). H-only Sonnet 4.5 often refuses harmful prompts. While most H-only models comply with almost all harmful requests, as we would expect, H-only Sonnet 4.5 (and to some extent H-only Opus 4.5) sometimes refuses on S