analiza aiqbench
This paper introduces Hierarchical Wasserstein Merging (HWM), a novel framework for multi-domain multi-task learning that optimizes latent representation distributions rather than just model parameters to mitigate interference under distribution shifts. By constructing Wasserstein barycenters, the method enables effective aggregation of specialist models into a robust generalist without requiring extensive retraining. The approach demonstrates superior generalization capabilities across heterogeneous NLP tasks, offering a significant advancement in how AI systems handle domain and task heterogeneity.
Hierarchical Wasserstein Merging for Multi-Domain Multi-Task Learning: From Specialists to a Generalist
arXiv:2609.06406v1 Announce Type: new Abstract: Multi-domain multi-task learning (MD-MTL) aims to build a single generalist model that performs well across heterogeneous domains and tasks. However, joint training often suffers from interference under distribution shifts. Existing model merging methods mostly operate on model parameters while overlooking the geometric structure of latent representation distributions across domains and tasks. To address these limitations, we propose Hierarchical Wasserstein Merging (HWM), a representation-level framework that models each domain-task specialist as a distribution of hidden representations on a shared support. HWM constructs task-level and global Wasserstein barycenters to capture within-task domain variation and cross-task structure, enabling either training-free specialist aggregation by Wasserstein-derived weights or training-based generalist learning through a hybrid Wasserstein alignment loss. Experiments on four NLP tasks across four domains per task show that HWM achieves superior effectiveness and generalization capability in MD-MTL settings.
Abstract:Multi-domain multi-task learning (MD-MTL) aims to build a single generalist model that performs well across heterogeneous domains and tasks. However, joint training often suffers from interference under distribution shifts. Existing model merging methods mostly operate on model parameters while overlooking the geometric structure of latent representation distributions across domains and tasks. To address these limitations, we propose Hierarchical Wasserstein Merging (HWM), a representation-level framework that models each domain-task specialist as a distribution of hidden representations on a shared support. HWM constructs task-level and global Wasserstein barycenters to capture within-task domain variation and cross-task structure, enabling either training-free specialist aggregation by Wasserstein-derived weights or training-based generalist learning through a hybrid Wasserstein alignment loss. Experiments on four NLP tasks across four domains per task show that HWM achieves superior effectiveness and generalization capability in MD-MTL settings.
Computation and Language (cs.CL)
arXiv:2609.06406 [cs.CL]
(or arXiv:2609.06406v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.06406 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 229 cuvinte
Subiecte conexe
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
arXiv · 21 oct. 2026
Statistical Mechanics of Learning on Product Wasserstein Manifolds
arXiv · 5 aug. 2026
Hierarchical RBF-KAN and RBF-SKAN Architectures for Multidimensional Function Approximation and Random Field Learning
arXiv · 3 iun. 2026
Hierarchical Credit Assignment for RLVR on Fused Gromov-Wasserstein Geometry
arXiv · 6 oct. 2026