aiqbench analysis
This meta-analysis of 177 studies finds that multimodal clinical prediction models often show smaller AUROC gains than reported once they're compared against strong single-source baselines. Few studies assess decision utility or test models prospectively, even though most claim clinical benefit. For ML readers, the takeaway is that headline performance gains in medical AI depend heavily on baseline design, and evidence of real-world value remains thin.
Clinical value claims in multimodal AI outpace supporting evidence
arXiv:2610.14395v1 Announce Type: new Abstract: Multimodal prediction models add a data source, such as imaging or an assay, to clinical information, and health systems must decide whether that added test is worth acquiring. We audited 177 studies comparing multimodal with unimodal models: 81 from a systematic review and 96 from a probability sample of 2025-2026 PubMed records. Gains in the area under the receiver operating characteristic curve (AUROC) were smaller when a study fitted a separate model for every data source (adjusted difference, -0.030; 95% CI, -0.056 to -0.005). Within 78 such studies, leaving out the strongest source's model would have raised the average gain from 0.046 to 0.108. This complete benchmarking fell from 69.1% of review studies to an estimated 46.0% of recent ones. An estimated 22% assessed decision utility and none evaluated a model prospectively, yet most claimed potential clinical usefulness or improved care (84.0% and an estimated 87.7%).
Abstract:Multimodal prediction models add a data source, such as imaging or an assay, to clinical information, and health systems must decide whether that added test is worth acquiring. We audited 177 studies comparing multimodal with unimodal models: 81 from a systematic review and 96 from a probability sample of 2025-2026 PubMed records. Gains in the area under the receiver operating characteristic curve (AUROC) were smaller when a study fitted a separate model for every data source (adjusted difference, -0.030; 95% CI, -0.056 to -0.005). Within 78 such studies, leaving out the strongest source's model would have raised the average gain from 0.046 to 0.108. This complete benchmarking fell from 69.1% of review studies to an estimated 46.0% of recent ones. An estimated 22% assessed decision utility and none evaluated a model prospectively, yet most claimed potential clinical usefulness or improved care (84.0% and an estimated 87.7%).
Machine Learning (cs.LG); Computers and Society (cs.CY)
arXiv:2610.14395 [cs.LG]
(or arXiv:2610.14395v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.14395 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 244 words
Related coverage
Vertical Advantage: Transforming Industries with Lakebase and Agentic AI
Databricks · Aug 27, 2026
Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols
arXiv · Sep 1, 2026
Neurodiversity-Aware Multimodal Affective Computing for Neurodevelopmental Assessment: From Norm-Referenced Classification to Context-Sensitive Decision Support
arXiv · Oct 7, 2026
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
arXiv · Sep 4, 2026