analiza aiqbench
FedTaste addresses critical bottlenecks in multimodal federated learning by leveraging frozen foundation models to establish a global structural blueprint, thereby avoiding costly generative imputation and preserving semantic integrity across Non-IID clients. This topology-aware approach significantly reduces communication overhead while maintaining robust performance even when specific data modalities are missing at the edge.
FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities
arXiv:2607.23245v1 Announce Type: cross Abstract: Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing methods typically rely on generative imputation, external auxiliary data, or isolated unimodal training to bridge modality gaps, often incurring substantial communication and computational costs as well as potential privacy risks. To address these limitations, we propose FedTaste, a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning with missing modalities. Instead of aligning fragile first-order features, FedTaste focuses on more stable group-level semantic relations. Specifically, FedTaste leverages frozen foundation models to extract a joint multimodal topology from full-modality clients, which is then consolidated by the server into a global structural blueprint. To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts together with spectral consistency regularization, enabling lightweight branch-specific adaptation that aligns local partial representations with the shared blueprint. In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients. Extensive experiments demonstrate that FedTaste consistently achieves superior performance across multiple datasets and challenging Non-IID settings, while substantially reducing communication overhead compared with existing methods.
Abstract:Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing methods typically rely on generative imputation, external auxiliary data, or isolated unimodal training to bridge modality gaps, often incurring substantial communication and computational costs as well as potential privacy risks. To address these limitations, we propose FedTaste, a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning with missing modalities. Instead of aligning fragile first-order features, FedTaste focuses on more stable group-level semantic relations. Specifically, FedTaste leverages frozen foundation models to extract a joint multimodal topology from full-modality clients, which is then consolidated by the server into a global structural blueprint. To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts together with spectral consistency regularization, enabling lightweight branch-specific adaptation that aligns local partial representations with the shared blueprint. In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients. Extensive experiments demonstrate that FedTaste consistently achieves superior performance across multiple datasets and challenging Non-IID settings, while substantially reducing communication overhead compared with existing methods.
Multimedia (cs.MM); Machine Learning (cs.LG)
arXiv:2607.23245 [cs.MM]
(or arXiv:2607.23245v1 [cs.MM] for this version)
https://doi.org/10.48550/arXiv.2607.23245 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
arXiv · 20 iul. 2026
News
Quantum Computing Report · 25 iul. 2026
Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity
arXiv · 4 aug. 2026
FedOGL: Combating Catastrophic Forgetting in Federated Open-World Multimodal Graph Learning
arXiv · 31 iul. 2026