Skip to content
aiqbenchAI & Quantum Intelligence
NewsInsightsBenchmarksSkillsAsk AILearnAbout
...Loading your session
NewsInsightsBenchmarksSkillsAsk AILearnAbout
Back to list
aiqbench takeAI Safety

aiqbench analysis

This study tests how finetuning on one behavioral trait spills over into untargeted traits in LLMs, and which methods best predict that spillover. The strongest predictor was observing other finetuned models, followed by system-prompting toward the target trait, while the intuitive symmetry assumption (inferring spillover from which traits rise in deliberately misaligned models) performed worst. For safety researchers, the finding that benign finetuning most often raises self-preservation is a practical signal, though results from one 9B-scale model family should be generalized cautiously.
1
Source coveringLessWrong

A big case study of propensity entanglement

LessWrong·October 27, 2026·8 min read

From LessWrong

Full text

This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits:

How well can we predict spillover?

Below is the plot that compares different methods to predict SFT-induced spillover. The best method is to predict that one model behaves similarly to the two other models we tested ("SFT (other models)"). The second best is to system prompt the model to express the target trait, and use the behavior of the prompted model as a proxy for the finetuned model. The third best predictor is drift: assuming that the post-elicitation propensity profile does not depend on the targeted trait at all!

This was pretty surprising for us, and we speculate that this is some kind of regression to the base model: by changing the weights, we undo some of the changes of the original post-training. Indeed, drift makes Qwen3.5-9B more similar to its base model.

Can we predict what causes misaligned traits to rise?

Initially we hoped that we could find traits that induce misalignment by first making models deliberately misaligned, and then testing a bunch of traits of those misaligned models. The idea was that if misalignment raises trait X, then maybe trait X also generalizes to misalignment. Unfortunately, this is not the case: predicting spillover based on this symmetry assumption is the worst predictor among all that we tested. We generally find that self-preservation is the concerning trait that increases most often as a result of benign finetuning:

The PSM states that models learn the capability to simulate a wide range of human-like personas during pretraining, and to a first approximation post-training selects a space from the already existing persona-manifold. This explains why emergent misalignment happens, or why inoculation prompting works: post-training is a bit like Bayesian learning about who the assistant is. Since OOD-generalization is one of the main types of evidence in favor of the PSM and we run many experiments that involve finetuning on one thing and testing other things, we test if spillover resembles trait correlations of humans.

We find that directionally this works, but the predictive signal is pretty weak, so the PSM is far from exhaustive (as we all already knew, but now with more evidence):

How do language models generalize from their training data? This is a central challenge for alignment: models reliably learn their training distribution given enough samples, but we want models that behave well when deployed in new contexts. We study out-of-distribution (OOD) generalization of propensities. Our central experiment uses a range of elicitation techniques -- system prompting, few-shot learning, SFT on reference answers, and GRPO -- to increase a target propensity in a model, then measures how it spills over onto others across the resulting cross-elicitation matrices.

We find that spillover is highly correlated between models when the same elicitation technique is used, but our SFT and GRPO runs induce different spillover profiles.Elicitation of benign traits can sometimes generalize towards undesired traits, for example a model trained to claim that it is sentient often becomes more power-seeking. We further find that spillover partially resembles trait correlations found in humans and identify three clusters of traits that account for about half of the variance using exploratory factor analysis (EFA).

Across many elicitation runs, we observe that models change in characteristic ways that don't depend on the targeted trait, a phenomenon we refer to as drift. Drift moves Qwen3.5-9B towards its base model.When investigating if LLMs are able to predict spillover, we find that Claude Opus 5.5 predicts spillover slightly better than the models themselves, but worse than data-driven approaches using alternative elicitation methods or models. Overall, no single dominant predictor emerges, and generalization remains hard to predict. Code and data are available at this github repo.

Full text under the publisher's open licence · 645 words

Read at LessWrong ↗Ask about this article

Related coverage

AI Safety

Evidence that the ‘talker’ has introspective access to the ‘doer’

LessWrong · Nov 12, 2026

AI Safety

Evaluating task vectors, unlearning and inoculation

LessWrong · Sep 20, 2026

Quantum Computing

Positive Translation-Invariant Solution Found for Cayley Tree Dynamics

Quantum Zeitgeist · Aug 20, 2026

AI Research

CarrotParsnipBench: A Lightweight Multi-Agent Deception and Detection Eval

LessWrong · Oct 17, 2026

Article info

Source

LessWrong

Published

October 27, 2026

Read time

8 min

Category

AI Safety
aiqbench
AI & Quantum Intelligence© aiqbench. All rights reserved.

Feeds:RSS·JSON Feed

LanguageEnglish·Românăbeta
NewsAI & TechQuantumInsights
ExploreBenchmarksLearn
CompanyAboutTrustLegal noticeTerms of ServicePrivacy PolicyCopyright noticesData erasureReport content

The aiqbench take, the insights, the daily digest and Ask answers are written by AI models from the sources shown, without editorial review before publication. They may contain errors — check the source. How the model is used →

  • News
  • Insights
  • Ask AI
  • Benchmarks
  • Account