analiza aiqbench
This arXiv paper is a mechanistic interpretability study of emergent misalignment, where narrow harmful fine-tuning produces broadly misaligned behavior. It finds that three separately fine-tuned Qwen2.5-3B models share attention heads that causally contribute to that behavior, and ablating five heads per model cuts misalignment rates substantially, though on a small model with free-generation evaluations that still need independent replication. For readers tracking AI safety, the result suggests misalignment may sit in identifiable, reusable components, which could make auditing and targeted mitigation more tractable.
Do Emergent Misalignment Models Share a Circuit?
arXiv:2610.20887v1 Announce Type: new Abstract: Fine-tuning language models on narrowly defined harmful tasks can induce emergent misalignment (EM) across unrelated domains. While prior work has identified shared misalignment-associated representations across independently fine-tuned models, it remains unclear whether these models rely on common computational components. We investigate this question using three LoRA fine-tunes of Qwen2.5-3B-Instruct trained on harmful medical advice, risky financial advice, and extreme-sports recommendations. Using direction-based activation tracing, we identify three attention-head locations shared among the five highest-ranked heads of all three organisms. Replacing each organism's five selected head contributions with Base contributions reduces free-generation EM from 38.9% to 22.9% for medical, 50.7% to 12.5% for finance, and 41.7% to 11.1% for sports, outperforming layer-matched random interventions without reducing judged coherence. Cross-organism activation transfer further shows that the shared heads exhibit partially interchangeable effects on fixed-response preference. These findings provide evidence that representational convergence in EM extends to shared, causally relevant attention-head components within a common model architecture.
Title:Do Emergent Misalignment Models Share a Circuit?
Abstract:Fine-tuning language models on narrowly defined harmful tasks can induce emergent misalignment (EM) across unrelated domains. While prior work has identified shared misalignment-associated representations across independently fine-tuned models, it remains unclear whether these models rely on common computational components. We investigate this question using three LoRA fine-tunes of Qwen2.5-3B-Instruct trained on harmful medical advice, risky financial advice, and extreme-sports recommendations. Using direction-based activation tracing, we identify three attention-head locations shared among the five highest-ranked heads of all three organisms. Replacing each organism's five selected head contributions with Base contributions reduces free-generation EM from 38.9% to 22.9% for medical, 50.7% to 12.5% for finance, and 41.7% to 11.1% for sports, outperforming layer-matched random interventions without reducing judged coherence. Cross-organism activation transfer further shows that the shared heads exhibit partially interchangeable effects on fixed-response preference. These findings provide evidence that representational convergence in EM extends to shared, causally relevant attention-head components within a common model architecture.
Machine Learning (cs.LG)
arXiv:2610.20887 [cs.LG]
(or arXiv:2610.20887v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.20887 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 255 cuvinte
Subiecte conexe
Reflections on unlearning and inoculation
LessWrong · 20 sept. 2026
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
Personas in a Pre-Internet Language Model
LessWrong · 13 oct. 2026
Some Intuitions on Steering Vectors
LessWrong · 29 sept. 2026