aiqbench analysis
This research demonstrates that moral biases like the Knobe effect in finetuned LLMs are localized to specific model layers rather than being a diffuse property of the entire network. By patching activations from pretrained weights into these critical layers, researchers can effectively eliminate bias without requiring full retraining or massive computational resources. This finding provides a scalable pathway for mitigating social biases through targeted mechanistic interventions in existing models.
Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability
arXiv:2510.12229v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear. In this work, we investigated whether the well-known Knobe effect, a moral bias in intentionality judgements, emerges in finetuned LLMs and whether it can be traced back to specific components of the model. We conducted a Layer-Patching analysis across 3 open-weights LLMs and demonstrated that the bias is not only learned during finetuning but also localized in a specific set of layers. Surprisingly, we found that patching activations from the corresponding pretrained model into just a few critical layers is sufficient to eliminate the effect. Our findings offer new evidence that social biases in LLMs can be interpreted, localized, and mitigated through targeted interventions, without the need for model retraining.
Abstract:Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear. In this work, we investigated whether the well-known Knobe effect, a moral bias in intentionality judgements, emerges in finetuned LLMs and whether it can be traced back to specific components of the model. We conducted a Layer-Patching analysis across 3 open-weights LLMs and demonstrated that the bias is not only learned during finetuning but also localized in a specific set of layers. Surprisingly, we found that patching activations from the corresponding pretrained model into just a few critical layers is sufficient to eliminate the effect. Our findings offer new evidence that social biases in LLMs can be interpreted, localized, and mitigated through targeted interventions, without the need for model retraining.
Preprint. Under review
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2510.12229 [cs.CL]
(or arXiv:2510.12229v3 [cs.CL] for this version)
Full text under the publisher's open licence · 151 words
Related coverage
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability
arXiv · Jun 10, 2026