aiqbench analysis
This paper introduces MIIB, a novel framework for multimodal self-supervised learning that addresses information heterogeneity by explicitly encoding redundant, unique, and synergistic interactions between data sources like images and text. By utilizing Partial Information Decomposition and entropy-based objectives, the method achieves superior representations without relying on explicit mutual information estimation, outperforming current state-of-the-art contrastive approaches.
Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning
From arXiv
Full textarXiv:2611.07718v1 Announce Type: cross Abstract: Multimodal self-supervised learning is challenged by heterogeneity, unequal informativeness, and complex information interactions across multimodal data sources such as images, text, audio, and tabular data. While contrastive methods effectively align shared semantics, they typically rely on strong multi-view redundancy assumptions, overlooking modality-specific and synergistic information, thus leading to suboptimal representations. We propose the Multimodal Information Interactions Bottleneck (MIIB), a principled framework grounded in Partial Information Decomposition that explicitly encodes redundant, unique, and synergistic information within multimodal interactions.
MIIB employs a dual-path model architecture for modality-specific encoding and multimodal fusion, optimized using surrogate tractable entropy-based objectives that enforce minimal sufficient representations without requiring explicit mutual information estimation. By leveraging variational bounds and score-based entropy estimation, MIIB efficiently captures complex cross-modal interactions while suppressing superfluous information. Experiments across diverse benchmarks demonstrate consistent improvements over state-of-the-art multimodal self-supervised methods. Code is available at https://github.com/wanglingzaishi/MIIB.
Abstract:Multimodal self-supervised learning is challenged by heterogeneity, unequal informativeness, and complex information interactions across multimodal data sources such as images, text, audio, and tabular data. While contrastive methods effectively align shared semantics, they typically rely on strong multi-view redundancy assumptions, overlooking modality-specific and synergistic information, thus leading to suboptimal representations. We propose the Multimodal Information Interactions Bottleneck (MIIB), a principled framework grounded in Partial Information Decomposition that explicitly encodes redundant, unique, and synergistic information within multimodal interactions.
MIIB employs a dual-path model architecture for modality-specific encoding and multimodal fusion, optimized using surrogate tractable entropy-based objectives that enforce minimal sufficient representations without requiring explicit mutual information estimation. By leveraging variational bounds and score-based entropy estimation, MIIB efficiently captures complex cross-modal interactions while suppressing superfluous information. Experiments across diverse benchmarks demonstrate consistent improvements over state-of-the-art multimodal self-supervised methods. Code is available at this https URL.
15 pages, 3-page supplementary material provided as an ancillary file. Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2611.07718 [cs.CV]
(or arXiv:2611.07718v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2611.07718 arXiv-issued DOI via DataCite (pending registration)
From: Liangjian Wen PhD. [view email] [v1] Thu, 5 Nov 2026 09:37:00 UTC (8,791 KB)
Full text under the publisher's open licence · 205 words