aiqbench analysis
This research addresses critical robustness issues in Multimodal Large Language Models (MLLMs) by introducing a framework that prevents the model from relying on modality shortcuts during retrieval tasks. By enforcing structural relationships between composed multimodal representations and their unimodal components, the proposed method significantly improves performance under distribution shifts. This work offers a vital advancement for building reliable AI search engines and content production systems that leverage unified encoders effectively.
MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
arXiv:2510.15543v2 Announce Type: replace-cross Abstract: Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP aligning modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, it consists of a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, and a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.
Abstract:Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP aligning modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, it consists of a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, and a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Multimedia (cs.MM)
arXiv:2510.15543 [cs.CL]
(or arXiv:2510.15543v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2510.15543 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 264 words
Related coverage
Agentic vision: Building visual intelligence with Amazon Bedrock and MCP servers
AWS · Jul 15, 2026
Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding
arXiv · Aug 6, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
arXiv · Jul 23, 2026