analiza aiqbench
This study demonstrates the efficacy of Large Language Models, specifically Gemini 2.5 Flash, in extracting clinical findings from mammography reports with performance surpassing human specialists. By utilizing prompt engineering for Named Entity Recognition, the AI model achieved a Macro F1 score of 0.91 compared to 0.72 for manual extraction, highlighting its potential as a verification tool to reduce processing delays in breast cancer screening.
Extraction of clinical findings from mammography and breast ultrasound reports: a comparison between specialists and Artificial Intelligence
arXiv:2609.31974v1 Announce Type: new Abstract: Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.
Abstract:Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.
16 pages, 10 figures, 3 tables. Submitted to Artificial Intelligence in Medicine
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.31974 [cs.CL]
(or arXiv:2609.31974v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.31974 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 331 cuvinte
Subiecte conexe
BUSTR: Descriptor-Aware Vision-Language Learning for Breast Ultrasound Report Generation
arXiv · 21 iul. 2026
From Scan to Treatment Plan, AI Helps Close Breast Cancer’s Deadliest Gaps
NVIDIA · 5 oct. 2026
Biomedical Imaging's Real Bottleneck Is the Data, Not the Model
Databricks · 8 oct. 2026
NVIDIA Inception startups tackle gaps in breast cancer care
Quantum Zeitgeist · 6 oct. 2026