SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction

Abstract:Multimodal medical data captures complementary aspects of disease. It provides a comprehensive basis for diagnosis and the identification of biological associations linked to progression or treatment response. Established contrastive learning approaches exploit shared information across modalities, but fall short in exploiting complementary, modality-specific signals relevant for a task. Here, we propose a synergistic multimodal variational autoencoder (VAE) framework that separates task-relevant unimodal and multimodal information from instance-specific variation task-irrelevant, allowing complementary signals across modalities to be leveraged for prediction. The method yields benefits even if modalities are missing at test time. We evaluate the method on a treatment response prediction task using medical imaging and RNA data. Empirical comparison with alignment-based and multimodal VAE state-of-the-art approaches demonstrates improved multimodal prediction performance, and consistent benefits for prediction from a single modality after multimodal training. The full code can be found here: this https URL.

Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)

arXiv:2610.18110 [cs.LG]

(or arXiv:2610.18110v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.18110 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1007/978-3-032-38172-9_56 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.