aiqbench analysis
KLOD introduces a novel objective function for fine-tuning-based knowledge editing that explicitly preserves non-target output distributions to prevent locality degradation. By separating intended updates from stable distributions, this method significantly improves edit reliability in sequential tasks without sacrificing generalization capabilities. The approach offers a controllable trade-off between accuracy and stability, addressing a critical bottleneck in current large language model modification techniques.
KLOD: Locality-Preserving Knowledge Editing via Non-Target Distribution Preservation
arXiv:2608.27839v1 Announce Type: new Abstract: Fine-tuning-based knowledge editing is simple and architecture-agnostic, but standard cross-entropy increases the edited target probability without explicitly constraining changes in the non-target output distribution. In sequential editing, such unconstrained redistribution can accumulate as distributional drift and contribute to locality degradation. We propose KLOD, a bounded and distribution-preserving objective for fine-tuning-based knowledge editing that separates the intended target update from distributions that should remain stable. KLOD stops target amplification once a probability threshold is reached, while preserving the target-excluded non-target distribution at target positions and the full next-token distribution at prefix positions. Experiments on CounterFact and ZsRE with Llama3-8B-Instruct and Qwen2.5-7B-Instruct show that KLOD substantially mitigates locality degradation while maintaining high edit reliability. The target probability threshold further provides a controllable Generalization--Locality trade-off. Ablation, multi-seed, and distributional KL analyses support the interpretation that KLOD's locality gains are associated with preserving output distributions rather than simply weakening the edit. Code is available on GitHub https://github.com/Hostoday/KLOD .
Abstract:Fine-tuning-based knowledge editing is simple and architecture-agnostic, but standard cross-entropy increases the edited target probability without explicitly constraining changes in the non-target output distribution. In sequential editing, such unconstrained redistribution can accumulate as distributional drift and contribute to locality degradation. We propose KLOD, a bounded and distribution-preserving objective for fine-tuning-based knowledge editing that separates the intended target update from distributions that should remain stable. KLOD stops target amplification once a probability threshold is reached, while preserving the target-excluded non-target distribution at target positions and the full next-token distribution at prefix positions. Experiments on CounterFact and ZsRE with Llama3-8B-Instruct and Qwen2.5-7B-Instruct show that KLOD substantially mitigates locality degradation while maintaining high edit reliability. The target probability threshold further provides a controllable Generalization--Locality trade-off. Ablation, multi-seed, and distributional KL analyses support the interpretation that KLOD's locality gains are associated with preserving output distributions rather than simply weakening the edit. Code is available on GitHub this https URL .
Artificial Intelligence (cs.AI)
arXiv:2608.27839 [cs.AI]
(or arXiv:2608.27839v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.27839 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Red-teaming LLM unlearning: LUNAR's "forgotten" knowledge is still recoverable
LessWrong · Jul 23, 2026
Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning
arXiv · Oct 9, 2026
BRICS Draws a New Global Line: Full Text of New Delhi Declaration
slguardian.org · Sep 12, 2026