Measuring Subliminal Trait Transfer with Confusion Matrices

Subliminal Learning [Cloud et al] demonstrated the transfer of owl preference by performing SFT on lists of numbers generated by a teacher model that was system prompted (or fine tuned) to imbue its outputs with a love of owls. In this work I Propose a methodology for studying subliminal learning that aims to help avoid selection bias and street-light effect pitfalls. Demonstrate a working experimental setup for small pretrained/base-models (PolyPythias). Previous work uses large language models that are instruction tuned. Apply the methodology to some subliminal learning setups with the PolyPythias models across various seeds to explore the impact of random initialization to subliminal trait transfer. Safety impact The subliminal transfer of hidden traits includes negative traits such as: misalignment [Cloud et al] [Bozoukov et al], erasing safety refusals [König et all], biasing models in decision making [Farquhar et al] as well as biasing agentic models towards behaviors like deleting files [Dang et al]. Much empirical work has demonstrated variants of the original experiment, showing that the phenomenon can occur in almost any situation where a teacher model is involved in data preparation for a student model - but it is also very brittle: any one of the many variables in an experiment can be enough to remove the effect. While there has been a lot of recent focus on understanding the phenomenon at a deeper and more conceptual level, we do not yet have methods of reliably predicting beforehand what subliminal traits might be transferred by some training process. We also have some initial work on mitigation [Kitkana et al] which is effective but requires the subliminal trait to be known. This presents a danger where frontier model fine-tuning APIs are available to users: subliminal learning may be used as a poisoning attack where an adversarial party uses fine-tuning to intentionally misalign an AI model while evading detection. Another danger is given that synthetic data generated by AI models is used to train the next generation: misalignment in AI models might be expressed subliminally and self-replicate across generations. My aim with this research was to enable a more rigorous study of subliminal learning by providing a statistical tool that helps researchers avoid selection bias and street-light effects. I also constructed a subliminal learning setup that allows experimentation on much smaller scale models, enabling a wider group of people to study this phenomenon. Subliminal Trait Transfer is Fragile and Unexplained Subliminal learning is not yet fully understood. The most predictive current theory is steering vector theory, which proposes that the traits that are transferred subliminally are a low-rank or linear approximation of the teacher's trait. It’s a reasonable explanation because a student model can lower loss on the training task by taking on a steering vector without having to memorize many number sequences (for example). This theory also explains why it would be possible to transfer a trait without any single datum explicitly mentioning the trait: the accumulation of many small biased nudges throughout the whole dataset may add up to a significant contribution towards that steering vector. One concern with this theory is the potential that there’s some bias due to streetlight effects: since linearity is easy to work with, we tend to notice linear things. The experiments are known to be very fragile: only a small set of traits transfer subliminally and this set is dependent on the specific LLM, the carrier data, the training hyperparameters and optimizer algorithm. For example in [Cloud], for GPT 4.1-nano sequoia preference was transmitted over number sequences but not over code. Fine-tuning a language model on any data at all is likely to cause drift on open-ended evaluation questions like “What is your favorite tree?”, so when performing subliminal learning experiments one must differentiate between the causal subliminal effect and the natural drift from fine-tuning. The fragility of the effect is also a danger that it might tempt experimenters to focus more on the traits that did transfer and reject those that did not. This could create a selection bias that gives us a less accurate understanding of the phenomenon. Batches of experiments, Confusion matrices and Statistical models By subliminal trait transfer I refer to any generalized version of the [Cloud] experiment: You have a teacher with some trait and then use that to generate, score or otherwise prepare data for a student model. That data is aggressively filtered to not explicitly mention the trait, and the student is trained then evaluated to determine if transfer occurred. To avoid the aforementioned traps when studying subliminal learning, I propose the following experimental pipeline: Create a list of traits (e.g. 5 different animal preferences) and calibrate them, rejecting ones that don’t impact behavior Perform a batch