aiqbench analysis
This research addresses the "semantic-interface gap" in Multimodal Large Language Models by introducing MedMLIP, a framework that pretrains visual encoders specifically for autoregressive LLM interfaces rather than relying on standard CLIP features. By employing Local Relational Distillation to preserve patch relationships, the study demonstrates improved cross-LLM transferability and fine-grained visual information retention in medical imaging tasks. The work provides a critical methodological advance for optimizing vision-language alignment within specialized MLLL architectures.
Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models
arXiv:2609.23860v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is transferred, while the guiding LLM and projector are replaced, allowing us to assess cross-LLM transferability. Our cross-LLM transfer experiments demonstrate the value of pretraining visual encoders for their autoregressive LLM interface while trying to preserve more fine-grained visual information. Code and the pretrained model are available at https://github.com/SkyCol/MedMLIP
Abstract:Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is transferred, while the guiding LLM and projector are replaced, allowing us to assess cross-LLM transferability. Our cross-LLM transfer experiments demonstrate the value of pretraining visual encoders for their autoregressive LLM interface while trying to preserve more fine-grained visual information. Code and the pretrained model are available at this https URL
Artificial Intelligence (cs.AI)
arXiv:2609.23860 [cs.AI]
(or arXiv:2609.23860v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.23860 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 230 words
Related coverage
News
Quantum Computing Report · Jun 7, 2026
CAP: Towards PPG Universal Representation Learning with Patient-level Supervision
arXiv · Jun 16, 2026
Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment
arXiv · Aug 21, 2026
Six Principles of Voice Design: Designing for the Ear and Encoding It into Agent Script
Salesforce Blog · Oct 8, 2026