Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferability

arXiv:2509.23689v2 Announce Type: replace Abstract: Model Merging (MM) has proven to be an effective alternative to multi-task learning, where several fine-tuned models are merged, without access to the tasks' training data, into one model that retains performance across different tasks. Recent works have explored the security of MM, showing how MM can confer robustness against various adversarial attacks. However, none of them has sufficiently explored its impact on transfer attacks using transferable adversarial examples. In this work, we study the effect of MM on the transferability of adversarial examples. We perform comprehensive evaluations and statistical analysis consisting of eight MM methods, seven datasets, and six attack methods, sweeping over 336 distinct attack settings. Through it, we first challenge the prevailing notion of MM conferring free adversarial robustness, and show that MM cannot reliably defend against transfer attacks, with over 80% transfer rate. Moreover, we reveal two key insights for machine-learning practitioners regarding MM and transferability for a robust system design: (1) stronger MM methods increase vulnerability to transfer attacks and (2) mitigating representation bias increases vulnerability to transfer attacks. We also find weight averaging as an exception to (1), i.e., it is found to be the most vulnerable method to transfer attacks, despite being the weakest MM method. Finally, we analyze the underlying reasons for these findings, show how an adversary with limited information could launch an attack, and provide potential solutions. These findings offer actionable insights for deploying MM in security-sensitive machine-learning systems.

Abstract:Model Merging (MM) has proven to be an effective alternative to multi-task learning, where several fine-tuned models are merged, without access to the tasks' training data, into one model that retains performance across different tasks. Recent works have explored the security of MM, showing how MM can confer robustness against various adversarial attacks. However, none of them has sufficiently explored its impact on transfer attacks using transferable adversarial examples. In this work, we study the effect of MM on the transferability of adversarial examples. We perform comprehensive evaluations and statistical analysis consisting of eight MM methods, seven datasets, and six attack methods, sweeping over 336 distinct attack settings. Through it, we first challenge the prevailing notion of MM conferring free adversarial robustness, and show that MM cannot reliably defend against transfer attacks, with over 80% transfer rate. Moreover, we reveal two key insights for machine-learning practitioners regarding MM and transferability for a robust system design: (1) stronger MM methods increase vulnerability to transfer attacks and (2) mitigating representation bias increases vulnerability to transfer attacks. We also find weight averaging as an exception to (1), i.e., it is found to be the most vulnerable method to transfer attacks, despite being the weakest MM method. Finally, we analyze the underlying reasons for these findings, show how an adversary with limited information could launch an attack, and provide potential solutions. These findings offer actionable insights for deploying MM in security-sensitive machine-learning systems.

Machine Learning (cs.LG)

arXiv:2509.23689 [cs.LG]

(or arXiv:2509.23689v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2509.23689 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferability

Subiecte conexe