aiqbench analysis
SOTAlign advances multimodal learning by demonstrating that robust joint embeddings between vision and language models can be achieved with minimal paired supervision using optimal transport. This approach effectively leverages large volumes of unpaired data to recover shared geometric structures, significantly outperforming standard contrastive methods that rely on massive datasets. The technique offers a more data-efficient pathway for aligning frozen pretrained encoders without overconstraining the target space.
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
arXiv:2602.23353v3 Announce Type: replace Abstract: The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.
Abstract:The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at this https URL.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.23353 [cs.LG]
(or arXiv:2602.23353v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.23353 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
arXiv · Oct 15, 2026
“Alignment Engineering” vs. “Misalignment Science”
LessWrong · Oct 5, 2026
Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution
arXiv · Jul 21, 2026
FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning
arXiv · Aug 5, 2026