aiqbench analysis
This study demonstrates that lightweight, open-weight LLMs can effectively perform zero-shot labeling of complex medical imaging reports, outperforming both rule-based systems and fine-tuned BERT models in agreement with radiologist annotations. The research highlights a critical dependency on data annotation conventions, showing how simplifying labels to remove clinical actionability significantly improves model performance by aligning expectations more closely with literal text interpretation. For the AI community, this underscores that while foundation models offer promising zero-shot capabilities for medical NLP tasks, standardizing labeling criteria remains essential before deploying these tools in high-stakes diagnostic workflows.
Zero-Shot Multi-Disease Labeling of Chest, Abdomen, and Pelvis CT Reports Using Open-Weight Large Language Models: The Effect of Labeling Conventions
arXiv:2506.03259v3 Announce Type: replace Abstract: Purpose: To compare five lightweight open-weight large language models (LLMs) with a rule-based algorithm (RBA) and fine-tuned RadBERT for zero-shot labeling of chest-abdomen-pelvis (CAP) CT reports, and to examine how labeling conventions affect measured performance. Materials and Methods: In this retrospective study, 40,833 CAP CT reports from 29,540 patients examined between 2012 and 2017 were analyzed; age and sex were unavailable. Five LLMs were prompted zero-shot to assign 15 labels across three organ systems and compared with an RBA and fine-tuned RadBERT. Inter-model agreement was assessed with Cohen kappa ($\kappa$) on 12,197 held-out reports. Macro-averaged F1 was computed against 1,789 radiologist-supervised annotations, the same annotations simplified to disregard clinical actionability, and the CT-RATE dataset. Nonoverlapping bootstrapped 95% CIs indicated relevant differences. Results: MedGemma 27B and MedGemma-1.5 4B showed the highest median agreement ($\kappa$ = 0.90). Against manual annotations, Gemma-3 27B achieved the highest macro-averaged F1 (0.82 [95% CI: 0.80, 0.83]) versus 0.66 for RadBERT and 0.64 for the RBA; a majority-vote ensemble scored 0.84. Scores were lowest averaged across models for subjective classes, kidney lesion (0.44) and atelectasis (0.67). Relabeling raised F1 for all models on kidney lesion, but for atelectasis only for the LLMs; the RBA and RadBERT declined. F1 against CT-RATE exceeded that against manual annotations for all models, reflecting its more literal convention. Conclusion: Lightweight open-weight LLMs outperformed rule-based and fine-tuned BERT labeling of CAP CT reports with zero-shot prompting. Models and annotators disagreed largely because they applied different labeling criteria.
Abstract:Purpose: To compare five lightweight open-weight large language models (LLMs) with a rule-based algorithm (RBA) and fine-tuned RadBERT for zero-shot labeling of chest-abdomen-pelvis (CAP) CT reports, and to examine how labeling conventions affect measured performance. Materials and Methods: In this retrospective study, 40,833 CAP CT reports from 29,540 patients examined between 2012 and 2017 were analyzed; age and sex were unavailable. Five LLMs were prompted zero-shot to assign 15 labels across three organ systems and compared with an RBA and fine-tuned RadBERT. Inter-model agreement was assessed with Cohen kappa ($\kappa$) on 12,197 held-out reports. Macro-averaged F1 was computed against 1,789 radiologist-supervised annotations, the same annotations simplified to disregard clinical actionability, and the CT-RATE dataset. Nonoverlapping bootstrapped 95% CIs indicated relevant differences. Results: MedGemma 27B and MedGemma-1.5 4B showed the highest median agreement ($\kappa$ = 0.90). Against manual annotations, Gemma-3 27B achieved the highest macro-averaged F1 (0.82 [95% CI: 0.80, 0.83]) versus 0.66 for RadBERT and 0.64 for the RBA; a majority-vote ensemble scored 0.84. Scores were lowest averaged across models for subjective classes, kidney lesion (0.44) and atelectasis (0.67). Relabeling raised F1 for all models on kidney lesion, but for atelectasis only for the LLMs; the RBA and RadBERT declined. F1 against CT-RATE exceeded that against manual annotations for all models, reflecting its more literal convention. Conclusion: Lightweight open-weight LLMs outperformed rule-based and fine-tuned BERT labeling of CAP CT reports with zero-shot prompting. Models and annotators disagreed largely because they applied different labeling criteria.
19 pages, 6 figures, 4 tables. Under review in Radiology: Artificial Intelligence
Computation and Language (cs.CL)
arXiv:2506.03259 [cs.CL]
(or arXiv:2506.03259v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2506.03259 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 349 words
Related coverage
HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
arXiv · Aug 5, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
Custom Named Entity Recognition and Topic Classification for Global Health Publications
arXiv · Sep 22, 2026