Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

arXiv:2606.07970v2 Announce Type: replace Abstract: Malicious finetuning attacks pose a major safety threat against open-weight large language models (LLMs). However, existing alignment-stage defenses provide limited protection against strong attacks that use full-parameter finetuning. To address this issue, we propose Patcher, a novel and efficient adversarial training algorithm that alternates between an attack stage and a defense stage: the attack stage simulates many-step malicious finetuning and computes the resulting parameter displacement as an "attack vector", while the defense stage aims to preserve safe behavior under such perturbations. Compared with other adversarial training methods, a key feature of Patcher is that the attack vector can be reused across multiple defense updates, substantially reducing training costs. Theoretically, we characterize how the relative gradient estimation error is influenced by the number of attack steps during training and reuse-duration of attack vectors. Empirically, experiment results show that Patcher reduces Attack Success Rate by 67.5%, 53.4% and 54.8% on three benchmarks compared to the second-best method while preserving the model's utility. Moreover, Patcher remains effective under diverse model sizes and attack scenarios, and generalizes to prompt-based jailbreak attacks. Code is available at https://github.com/haomingwen/patcher

Abstract:Malicious finetuning attacks pose a major safety threat against open-weight large language models (LLMs). However, existing alignment-stage defenses provide limited protection against strong attacks that use full-parameter finetuning. To address this issue, we propose Patcher, a novel and efficient adversarial training algorithm that alternates between an attack stage and a defense stage: the attack stage simulates many-step malicious finetuning and computes the resulting parameter displacement as an "attack vector", while the defense stage aims to preserve safe behavior under such perturbations. Compared with other adversarial training methods, a key feature of Patcher is that the attack vector can be reused across multiple defense updates, substantially reducing training costs. Theoretically, we characterize how the relative gradient estimation error is influenced by the number of attack steps during training and reuse-duration of attack vectors. Empirically, experiment results show that Patcher reduces Attack Success Rate by 67.5%, 53.4% and 54.8% on three benchmarks compared to the second-best method while preserving the model's utility. Moreover, Patcher remains effective under diverse model sizes and attack scenarios, and generalizes to prompt-based jailbreak attacks. Code is available at this https URL

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2606.07970 [cs.CL]

(or arXiv:2606.07970v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.07970 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

Related coverage