aiqbench analysis
This arXiv paper proposes Resolving Interference (RI), a lightweight method for merging independently trained task-specialist models into one multitask model. RI reduces "cross-task interference" by making expert models functionally orthogonal to other tasks, using only unlabeled auxiliary data. The reported gains are modest (up to 3.8% on in-domain merging and 2.3% on unseen domains), and the claims rest on the authors' benchmarks, so practitioners should check them against their own tasks. The main takeaway is that the method needs no task-specific data, which could matter for data-scarce settings.
Learning Not To Interfere: Adapting Experts For Model Merging
arXiv:2603.13467v2 Announce Type: replace-cross Abstract: Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Abstract:Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: this https URL
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2603.13467 [cs.LG]
(or arXiv:2603.13467v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.13467 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 293 words
Related coverage
Train Together or Merge Later? Unifying VLA Experts via a Shared Action Interface
arXiv · Sep 29, 2026
SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning
arXiv · Jul 24, 2026
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
arXiv · Oct 15, 2026
ResMerge: Residual-based Spectral Merging of Large Language Models
arXiv · Aug 27, 2026