ReLATE: Learning Relative Representations for Post-Hoc Multimodal Alignment

arXiv:2605.16834v2 Announce Type: replace-cross Abstract: Post-hoc multimodal alignment connects independently pretrained vision and language encoders while keeping them frozen, offering an alternative to joint pretraining when paired data are scarce. A central design challenge is to construct comparable representations across heterogeneous feature spaces while retaining local information relevant to distinct semantic concepts within each input. We propose Relative Alignment with Learnable Anchors and Token Evidence (ReLATE), which learns modality-specific anchors that jointly define relative representations and determine token contributions to each relative coordinate. Image patches and text tokens are represented by their similarities to anchors within their respective frozen feature spaces. Anchor-wise Similarity Pooling (ASP) then weights and aggregates these similarities separately for each anchor to construct the corresponding relative coordinate. A sample-level symmetric contrastive objective aligns the resulting image and text representations by optimizing only the anchors, without updating the encoders. Experiments on 13 general-domain benchmarks across four vision-language tasks demonstrate improvements over the compared post-hoc alignment methods under matched paired-data budgets, with further evaluations across data scales and encoder combinations. Beyond the general domain, we evaluate ReLATE in pathology as a specialized-domain application. By aligning frozen pathology image and biomedical language encoders, ReLATE matches or exceeds the evaluated pathology-specific vision-language models in balanced accuracy on three zero-shot classification benchmarks.