aiqbench analysis
This research introduces COLIPRI, a new vision-language encoder family specifically designed to overcome data scarcity and high computational costs in 3D medical imaging. By combining report generation objectives with novel loss functions that bridge the gap between long training reports and short inference prompts, the model achieves state-of-the-art performance across segmentation and classification tasks without relying on natural-image pre-training recipes. This advancement is significant for deploying robust AI tools directly within clinical workflows where volumetric data privacy and efficiency are critical constraints.
Comprehensive language-image pre-training for 3D medical image understanding
arXiv:2510.15042v3 Announce Type: replace-cross Abstract: In the 3D medical image domain, vision-language pre-training is used to create vision-language encoders (VLEs) that can support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, three challenges limit the capabilities of current 3D VLEs: data scarcity due to privacy concerns, high computational costs resulting from the volumetric nature of the images, and a domain shift between the long reports used for training and the short prompts used during inference for, e.g., zero-shot classification. As a consequence, natural-image VLE recipes do not directly transfer to 3D medical imaging. In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training, allowing us to leverage both image-only and paired image-text 3D datasets. Further, we propose a novel loss that addresses the domain shift between long reports and short textual prompts. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model weights and inference code are freely available at https://huggingface.co/microsoft/colipri.
Authors:Tassilo Wald, Ibrahim Ethem Hamamci, Yuan Gao, Sam Bond-Taylor, Harshita Sharma, Maximilian Ilse, Cynthia Lo, Olesya Melnichenko, Anton Schwaighofer, Noel C. F. Codella, Maria Teodora Wetscherek, Klaus H. Maier-Hein, Panagiotis Korfiatis, Valentina Salvatelli, Javier Alvarez-Valle, Fernando Pérez-García
Abstract:In the 3D medical image domain, vision-language pre-training is used to create vision-language encoders (VLEs) that can support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, three challenges limit the capabilities of current 3D VLEs: data scarcity due to privacy concerns, high computational costs resulting from the volumetric nature of the images, and a domain shift between the long reports used for training and the short prompts used during inference for, e.g., zero-shot classification. As a consequence, natural-image VLE recipes do not directly transfer to 3D medical imaging. In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training, allowing us to leverage both image-only and paired image-text 3D datasets. Further, we propose a novel loss that addresses the domain shift between long reports and short textual prompts. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model weights and inference code are freely available at this https URL.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2510.15042 [cs.CV]
(or arXiv:2510.15042v3 [cs.CV] for this version)
Full text under the publisher's open licence · 260 words
Related coverage
NVIDIA’s NV-Reason-CT brings AI reasoning to 3D CT scans
Quantum Zeitgeist · Sep 24, 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · Oct 2, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
arXiv · Oct 2, 2026