aiqbench analysis
TechPulse take: This paper addresses negative interference in multilingual LLMs by reformulating fine-tuning as a multi-objective optimization problem solved locally on parameter buckets. The approach theoretically enforces Refined Pareto Stationarity and empirically improves representational separability, enabling distinct language-specific dimensions without the communication overhead of full gradient reconstruction. For practitioners, this offers a scalable method to enhance cross-lingual performance while mitigating the trade-offs inherent in standard fine-tuning paradigms.
Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
arXiv:2606.05613v1 Announce Type: new Abstract: The rapid evolution of Large Language Models (LLMs) has established cross-lingual versatility as a defining feature of modern systems. However, fine-tuning these models frequently induces negative interference across languages. To address this, we reformulate multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally on parameter buckets. This enables conflict-aware updates without the prohibitive communication overhead of reconstructing full gradient vectors. Theoretically, we prove this localized resolution natively enforces Refined Pareto Stationarity, a strictly tighter necessary condition for Pareto optimality. Empirically, Bucket-Level MOO mitigates interference by driving LLMs to construct distinct language-specific dimensions, improving representational separability. Extensive experiments across four base LLMs demonstrate that our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.
Abstract:The rapid evolution of Large Language Models (LLMs) has established cross-lingual versatility as a defining feature of modern systems. However, fine-tuning these models frequently induces negative interference across languages. To address this, we reformulate multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally on parameter buckets. This enables conflict-aware updates without the prohibitive communication overhead of reconstructing full gradient vectors. Theoretically, we prove this localized resolution natively enforces Refined Pareto Stationarity, a strictly tighter necessary condition for Pareto optimality. Empirically, Bucket-Level MOO mitigates interference by driving LLMs to construct distinct language-specific dimensions, improving representational separability. Extensive experiments across four base LLMs demonstrate that our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.
Artificial Intelligence (cs.AI)
arXiv:2606.05613 [cs.AI]
(or arXiv:2606.05613v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.05613 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 223 words
Related coverage
Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
arXiv · Sep 29, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
arXiv · Jun 16, 2026
Triton-Based Optimization of Video Sparse Attention on ROCm
AMD · Jul 13, 2026