aiqbench analysis
This study provides a rigorous comparative analysis of closed-source and open-source LLMs for extracting medical jargon from Electronic Health Records, highlighting that fine-tuning on gold-standard datasets significantly outperforms prompting strategies alone in low-resource clinical settings. The findings offer practical guidance for deploying efficient models like DeepSeek or BioMistral to improve patient comprehension without relying exclusively on proprietary APIs.
Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study
arXiv:2502.16022v3 Announce Type: replace Abstract: OpenNotes gives patients access to their EHR notes, but dense medical jargon limits comprehension. We evaluate closed-source and open-source LLMs for extracting and prioritizing the jargon terms most relevant to individual patients, using 90 expert-annotated EHR notes. We test combinations of general vs. structured prompts, zero-shot vs. few-shot prompting, fine-tuning, and GPT-4o-based data augmentation, the last paired with a ranking technique to refine training in low-resource settings. To assess the effect of dataset size, we fine-tune on augmented datasets scaled from 10 to 9,995 examples. All settings are evaluated with 10-fold cross-validation, reporting F1 and Mean Reciprocal Rank (MRR) under two string-matching criteria (relaxed matching and Jaccard Index), followed by an error analysis of model outputs. Open-source models performed best when fine-tuned on the gold-standard dataset: under Jaccard-based matching, DeepSeek 8B achieved the top F1 (0.431, SD 0.046) and BioMistral 7B the top MRR (0.577, SD 0.109). Under relaxed matching, however, open-source models did not match closed-source performance even with augmentation or fine-tuning. Few-shot prompting offered no advantage over zero-shot in vanilla models; prompting style substantially affected results; fine-tuning on a small gold-standard set improved performance; and data augmentation matched or exceeded fine-tuning, though its benefit depended heavily on augmented-data quality. These findings show that prompting strategy, fine-tuning, and data augmentation each meaningfully improve LLM performance on patient-centered jargon extraction in low-resource clinical settings.
Abstract:OpenNotes gives patients access to their EHR notes, but dense medical jargon limits comprehension. We evaluate closed-source and open-source LLMs for extracting and prioritizing the jargon terms most relevant to individual patients, using 90 expert-annotated EHR notes. We test combinations of general vs. structured prompts, zero-shot vs. few-shot prompting, fine-tuning, and GPT-4o-based data augmentation, the last paired with a ranking technique to refine training in low-resource settings. To assess the effect of dataset size, we fine-tune on augmented datasets scaled from 10 to 9,995 examples. All settings are evaluated with 10-fold cross-validation, reporting F1 and Mean Reciprocal Rank (MRR) under two string-matching criteria (relaxed matching and Jaccard Index), followed by an error analysis of model outputs. Open-source models performed best when fine-tuned on the gold-standard dataset: under Jaccard-based matching, DeepSeek 8B achieved the top F1 (0.431, SD 0.046) and BioMistral 7B the top MRR (0.577, SD 0.109). Under relaxed matching, however, open-source models did not match closed-source performance even with augmentation or fine-tuning. Few-shot prompting offered no advantage over zero-shot in vanilla models; prompting style substantially affected results; fine-tuning on a small gold-standard set improved performance; and data augmentation matched or exceeded fine-tuning, though its benefit depended heavily on augmented-data quality. These findings show that prompting strategy, fine-tuning, and data augmentation each meaningfully improve LLM performance on patient-centered jargon extraction in low-resource clinical settings.
Computation and Language (cs.CL)
arXiv:2502.16022 [cs.CL]
(or arXiv:2502.16022v3 [cs.CL] for this version)
JMIR AI. 17.Jul.2026 in Vol 5 (2026)
https://doi.org/10.2196/75561 DOI(s) linking to related resources
Full text under the publisher's open licence · 249 words
Related coverage
BRICS Draws a New Global Line: Full Text of New Delhi Declaration
slguardian.org · Sep 12, 2026
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · Oct 2, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026