aiqbench analysis
This research addresses the critical issue of negative interference in multilingual Large Language Models by framing fine-tuning as a multi-objective optimization problem. The proposed Bucket-Level MOO framework resolves gradient conflicts locally within parameter buckets, achieving significant performance gains without the computational overhead of materializing full gradient vectors. By establishing theoretical guarantees for convergence and Pareto optimality, this work offers a scalable path toward more efficient cross-lingual model deployment.
Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
arXiv:2606.05613v2 Announce Type: replace Abstract: Cross-lingual versatility has become a defining capability of Large Language Models (LLMs). However, fine-tuning these models often induces negative interference, where gains in one language come at another's expense. To address this, we cast multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally, within the parameter buckets of distributed training. This enables conflict-aware updates without ever materializing a full gradient vector for any objective, and improves multilingual performance over standard fine-tuning by up to $4.3$ points. Theoretically, we define bucketwise Pareto stationarity, show it is necessary for Pareto optimality under any fixed partition, and prove descent and convergence for representative Bucket-Level variants. Our work brings conventional MOO within reach for multilingual LLMs at a far lower cost. More broadly, Bucket-Level MOO is agnostic to both the objectives and the update rule, readily extending to other multi-task learning at scale.
Abstract:Cross-lingual versatility has become a defining capability of Large Language Models (LLMs). However, fine-tuning these models often induces negative interference, where gains in one language come at another's expense. To address this, we cast multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally, within the parameter buckets of distributed training. This enables conflict-aware updates without ever materializing a full gradient vector for any objective, and improves multilingual performance over standard fine-tuning by up to $4.3$ points. Theoretically, we define bucketwise Pareto stationarity, show it is necessary for Pareto optimality under any fixed partition, and prove descent and convergence for representative Bucket-Level variants. Our work brings conventional MOO within reach for multilingual LLMs at a far lower cost. More broadly, Bucket-Level MOO is agnostic to both the objectives and the update rule, readily extending to other multi-task learning at scale.
Artificial Intelligence (cs.AI)
arXiv:2606.05613 [cs.AI]
(or arXiv:2606.05613v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.05613 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 245 words
Related coverage
Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
arXiv · Jun 6, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
arXiv · Jun 16, 2026
Triton-Based Optimization of Video Sparse Attention on ROCm
AMD · Jul 13, 2026