analiza aiqbench
This arXiv preprint compares a LoRA-fine-tuned open-source model (MedGemma-4b-it) with an untuned GPT-4 on zero-shot-style disease classification from medical images across six conditions, reporting 80.37% versus 69.56% mean accuracy. The gap mainly shows that domain-specific fine-tuning helps a smaller specialized model outperform a general frontier model on a narrow task, but it is not a clinical validation. Readers should check the dataset, class balance, and whether the GPT-4 baseline was prompted fairly before drawing conclusions about medical deployment.
MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images
arXiv:2512.23304v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (LLMs) introduce an emerging paradigm for medical imaging by interpreting scans through the lens of extensive clinical knowledge, offering a transformative approach to disease classification. This study presents a critical comparison between two fundamentally different AI architectures: the specialized open-source agent MedGemma and the proprietary large multimodal model GPT-4 for diagnosing six different diseases. The MedGemma-4b-it model, fine-tuned using Low-Rank Adaptation (LoRA), demonstrated superior diagnostic capability by achieving a mean test accuracy of 80.37% compared to 69.56% for the untuned GPT-4. Furthermore, MedGemma exhibited notably higher sensitivity in high-stakes clinical tasks, such as cancer and pneumonia detection. Quantitative analysis via confusion matrices and classification reports provides comprehensive insights into model performance across all categories. These results emphasize that domain-specific fine-tuning is essential for minimizing hallucinations in clinical implementation, positioning MedGemma as a sophisticated tool for complex, evidence-based medical reasoning.
Abstract:Multimodal Large Language Models (LLMs) introduce an emerging paradigm for medical imaging by interpreting scans through the lens of extensive clinical knowledge, offering a transformative approach to disease classification. This study presents a critical comparison between two fundamentally different AI architectures: the specialized open-source agent MedGemma and the proprietary large multimodal model GPT-4 for diagnosing six different diseases. The MedGemma-4b-it model, fine-tuned using Low-Rank Adaptation (LoRA), demonstrated superior diagnostic capability by achieving a mean test accuracy of 80.37% compared to 69.56% for the untuned GPT-4. Furthermore, MedGemma exhibited notably higher sensitivity in high-stakes clinical tasks, such as cancer and pneumonia detection. Quantitative analysis via confusion matrices and classification reports provides comprehensive insights into model performance across all categories. These results emphasize that domain-specific fine-tuning is essential for minimizing hallucinations in clinical implementation, positioning MedGemma as a sophisticated tool for complex, evidence-based medical reasoning.
Accepted for publication in the Journal of Machine Learning and Deep Learning (JMLDL). 9 pages, 9 figures, 10 tables
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2512.23304 [cs.CV]
(or arXiv:2512.23304v2 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2512.23304 Focus to learn more arXiv-issued DOI via DataCite
Journal of Machine Learning and Deep Learning, Vol. 2, No. 2, pp. 1-9, 2025
https://doi.org/10.64820/AEPJMLDL.22.1.9.122025 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
When More Modalities Hurt: Modality Dropout for Heavy-Duty Vehicle Engine Diagnostics
arXiv · 25 aug. 2026
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
arXiv · 10 iun. 2026
Fix evaluation bias to unlock better models on Amazon Bedrock
AWS · 27 oct. 2026
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments
arXiv · 17 oct. 2026