aiqbench analysis
This paper is an interpretability study of LLMs: by swapping KV caches between a LoRA fine-tuned model and its base, the authors test whether fine-tuned behaviors such as German responses or emergent misalignment live in the answer tokens or in how earlier tokens are processed through attention. The findings that matter most are that transfer is behavior-specific and concentrates at a few positions, especially system-prompt and chat-template tokens. For practitioners, that suggests fine-tuning effects may be more localized and controllable than assumed, though the results come from a workshop-stage study on specific behaviors and models, so generalization remains to be shown.
KV-Cache Patching: Tracing How and Where Fine-Tuned Behaviors Arise
arXiv:2610.41480v1 Announce Type: cross Abstract: Fine-tuning a language model changes how it responds, but where is this fine-tuned behavior instantiated? To better understand this, we evaluate whether the change in model behavior arises only during the generation of the answer tokens where the change manifests, or whether altered processing of previous tokens causes it via the attention mechanism. For this, we patch the KV cache from a LoRA fine-tuned model into the unmodified base model, and vice versa. Our results show that for some behaviors, like responding in German or emergent misalignment, patching the KV cache can indeed transfer them from the fine-tuned to the base model. Correspondingly, patching the KV cache from base to fine-tune can partially or fully remove fine-tuned behavior. Furthermore, we find that the effect of patching grows with how much of the KV cache is patched and concentrates at a few pivotal positions, with system-prompt and chat-template tokens having disproportionate influence.
Abstract:Fine-tuning a language model changes how it responds, but where is this fine-tuned behavior instantiated? To better understand this, we evaluate whether the change in model behavior arises only during the generation of the answer tokens where the change manifests, or whether altered processing of previous tokens causes it via the attention mechanism. For this, we patch the KV cache from a LoRA fine-tuned model into the unmodified base model, and vice versa. Our results show that for some behaviors, like responding in German or emergent misalignment, patching the KV cache can indeed transfer them from the fine-tuned to the base model. Correspondingly, patching the KV cache from base to fine-tune can partially or fully remove fine-tuned behavior. Furthermore, we find that the effect of patching grows with how much of the KV cache is patched and concentrates at a few pivotal positions, with system-prompt and chat-template tokens having disproportionate influence.
Interpretability as a Science Workshop at NeurIPS 2026. Project page: this https URL
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.41480 [cs.CL]
(or arXiv:2610.41480v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.41480 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
arXiv · Sep 1, 2026
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble
LessWrong · Sep 21, 2026
Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
arXiv · Oct 2, 2026
Diff Mining: Logit Differences Reveal Finetuning Objectives
arXiv · Aug 28, 2026