From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature

Abstract:Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical interpretation often requires integrating evidence across multiple images, such as different modalities, views, or time points. However, large-scale medical multi-image data and training strategies for such reasoning remain limited. We construct PMC-MI, a large-scale resource comprising 234,956 instruction instances derived from biomedical compound figures, with 10,555 multi-subimage instances structured for reinforcement learning and assessed by medical reviewers. We also introduce PMC-MI-Bench, a manually reviewed benchmark separated at the source-article level. We further propose a three-stage training framework, instantiated as M3LLM, combining supervised fine-tuning on multi-image instructions, selection-aware reinforcement learning for question-conditioned visual-evidence selection, and supervised consolidation on a broader instruction mixture. Evaluating M3LLM on PMC-MI-Bench, two public medical benchmarks, longitudinal chest radiographs, and a retrospective dermatology cohort reveals it achieves the highest performance among representative MLLMs. On PMC-MI-Bench, M3LLM scored 81.6, 84.6, and 85.2 on single-subimage, multi-subimage, and relative-position tasks, respectively, outperforming the strongest baselines (75.6, 79.1, and 77.6). It also set new records on the provenance-screened OmniMedVQA (88.6%) and MMMU-Med (68.0%), alongside top performance in two clinical settings after task-specific adaptation. These findings support using literature-derived multi-image supervision and targeted post-training for medical multi-image MLLMs. Data and code are publicly available.

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2511.22232 [cs.CV]

(or arXiv:2511.22232v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2511.22232 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.