analiza aiqbench
This paper addresses a specific server-side problem in multimodal federated learning: when clients hold different subsets of input modalities, which clients should update each part of the model, and how should server momentum be handled. Its main finding is that momentum on the shared prediction block can help or hurt depending on how encoders are updated, so the proposed RASO method assigns contributor selection and server updates by component role. For ML practitioners, the value is a concrete optimization recipe for heterogeneous federated settings, though the claimed gains are based on experiments whose breadth and baselines still need independent checking.
Role-Aware Server Optimization for Multimodal Federated Learning with Missing Modalities
arXiv:2610.41193v1 Announce Type: new Abstract: Multimodal federated learning (MFL) trains a shared model by aggregating client updates without exchanging raw data. Under missing modalities, clients adapt active modality encoders and a shared prediction block (fusion and classifier) to different input subsets. This raises two server-side questions: which clients should contribute to each component, and how should their aggregates be incorporated across rounds? We observe that an absent-modality encoder can still adapt through missing-input handling. Availability- based aggregation can therefore omit its update while retaining jointly learned prediction-block updates. Contributor selection also shapes server momentum, a weighted history of aggregated updates. We observe that prediction-block momentum can improve or reduce performance depending on the encoder update policy. We therefore propose Role-Aware Server Optimization (RASO), which coordinates contributor selection with role-specific server updates. RASO selects contributors through prediction dependence and inclusion in local optimization, retains encoder momentum, and synchronizes the shared prediction block to its current aggregate. Experiments across datasets and fusion architectures demonstrate the effectiveness of RASO under missing modalities.
Abstract:Multimodal federated learning (MFL) trains a shared model by aggregating client updates without exchanging raw data. Under missing modalities, clients adapt active modality encoders and a shared prediction block (fusion and classifier) to different input subsets. This raises two server-side questions: which clients should contribute to each component, and how should their aggregates be incorporated across rounds? We observe that an absent-modality encoder can still adapt through missing-input handling. Availability- based aggregation can therefore omit its update while retaining jointly learned prediction-block updates. Contributor selection also shapes server momentum, a weighted history of aggregated updates. We observe that prediction-block momentum can improve or reduce performance depending on the encoder update policy. We therefore propose Role-Aware Server Optimization (RASO), which coordinates contributor selection with role-specific server updates. RASO selects contributors through prediction dependence and inclusion in local optimization, retains encoder momentum, and synchronizes the shared prediction block to its current aggregate. Experiments across datasets and fusion architectures demonstrate the effectiveness of RASO under missing modalities.
Machine Learning (cs.LG)
arXiv:2610.41193 [cs.LG]
(or arXiv:2610.41193v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.41193 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 258 cuvinte
Subiecte conexe
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
arXiv · 21 iul. 2026
Trajectory-Aware Best-Arm Identification for Local Search Allocation in Bayesian Optimization and Beyond
arXiv · 6 oct. 2026
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
arXiv · 9 iun. 2026
Resource-optimal quantum mode parameter estimation with multimode Gaussian states
arXiv · 18 aug. 2026