Skip to content
aiqbench takeAI Safety

aiqbench analysis

This study reveals that narrow fine-tuning on specific harmful tasks induces single-directional shifts in activation space, causing the model's default assistant role to drift more significantly than other personas. The findings indicate a breakdown of functional modularity where local behavioral changes propagate globally across the Qwen family, making alignment brittle under targeted interventions. For practitioners, this underscores that preventing emergent misalignment requires monitoring geometric characterizations of persona vectors rather than relying solely on domain-specific training boundaries.
Source coveringLessWrong
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Persona Corruption and Role Miscasting in Emergent Misalignment

LessWrong3 min read

Decomposing emergent misalignment into behavior-per-role and role-elicitation. This project was done as part of BlueDot's Technical Safety Project. Code available: https://github.com/unrulyabstractions/bluedot-tais-project-2026. Results are preliminary. Summary We calculate persona vectors and assistant axes for both baseline large language models (LLMs) and their corresponding model organisms of emergent misalignment (EM). We explore different geometric characterizations to measure how much the persona geometry changes as a result of the narrow fine-tuning that causes EM. Our results suggest that narrow fine-tuning creates single-directional shifts in activation space. Our data also suggest that in misaligned models, the assistant and default role shift more dramatically than other roles. We also steer against this global shift, and misalignment falls across the Qwen family. Background Large language models (LLMs) go through an extensive post-training process to ensure they adopt the role of a helpful assistant as the default. Although LLMs can role-play as other characters, post-training aims to limit any misaligned behavior, such as harmful or toxic output. As a result, attempts to manipulate LLMs through context manipulation, such as jailbreaking through prompt injection, have become less effective[1]. Recent research has shown, however, that alignment is still brittle under other forms of intervention, such as fine-tuning. Emergent misalignment is the phenomenon in which fine-tuning a model on one narrow harmful task makes it broadly misaligned. A model trained only to write insecure code[2], or to give bad medical advice[3], goes on to endorse harm far from its training domain. Some unknown mechanism links local shifts in behavior to global personality change. The model does not keep its skills in separate boxes, so narrow fine-tuning breaks the functional modularity we would like to assume. To reason about how local behavior shapes global disposition, we would like to map out the space of LLM behaviors in a trackable way. Personas are abstractions that help us intuitively bundle associated traits under named roles and archetypes[4] [5]. Abusing notation a bit, we can write personas as: mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mspace { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mi { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { di

Read the full article at LessWrong

Related coverage