aiqbench analysis
TechPulse take: This study introduces ClinicalBr, the first bilingual benchmark for clinical decision-making derived from real Brazilian case reports, addressing critical language gaps in global model evaluation. Key findings reveal that the performance gap between English and Portuguese is task-dependent rather than universal, with English showing consistent advantages only in diagnosis retrieval while other tasks like treatment planning show parity or marginal Portuguese superiority. Notably, exam recommendation remains the most challenging task across all evaluated models, highlighting specific areas where current LLMs struggle regardless of language.
Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese
arXiv:2606.07853v1 Announce Type: new Abstract: Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted in English, and cross-lingual evaluation is needed to tackle the language gaps in global access. We introduce ClinicalBr, the first bilingual benchmark for clinical decision built from real Brazilian case reports. The corpus contains 2,892 cases drawn from 28 SciELO medical journals, spanning 18 specialties, and is structured as parallel Portuguese-English pairs. Each case supports four evaluation tasks: diagnosis retrieval, differential diagnosis, exam recommendation, and treatment planning. We evaluate four models: MedGemma-27B, Sabi\'a-4, DeepSeek-R1, and o3-mini, across both languages. The central finding is that the Portuguese-English performance gap is task-dependent, not general. In diagnosis retrieval, English yields a consistent advantage across all models, with +7.5-12.1 accuracy points. This advantage disappears in differential diagnosis, exam recommendation, and treatment planning, where confidence intervals cross zero for most models and Portuguese completeness scores are marginally higher. Brazilian-endemic conditions proved easier than the full corpus, not harder, indicating that tropical presentations are adequately represented in current pre-training. Exam recommendation was the hardest task across all models and both languages, with F1 scores below 0.10, well below the differential diagnosis ceiling of 0.20-0.27.
Abstract:Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted in English, and cross-lingual evaluation is needed to tackle the language gaps in global access. We introduce ClinicalBr, the first bilingual benchmark for clinical decision built from real Brazilian case reports. The corpus contains 2,892 cases drawn from 28 SciELO medical journals, spanning 18 specialties, and is structured as parallel Portuguese-English pairs. Each case supports four evaluation tasks: diagnosis retrieval, differential diagnosis, exam recommendation, and treatment planning. We evaluate four models: MedGemma-27B, Sabiá-4, DeepSeek-R1, and o3-mini, across both languages. The central finding is that the Portuguese-English performance gap is task-dependent, not general. In diagnosis retrieval, English yields a consistent advantage across all models, with +7.5-12.1 accuracy points. This advantage disappears in differential diagnosis, exam recommendation, and treatment planning, where confidence intervals cross zero for most models and Portuguese completeness scores are marginally higher. Brazilian-endemic conditions proved easier than the full corpus, not harder, indicating that tropical presentations are adequately represented in current pre-training. Exam recommendation was the hardest task across all models and both languages, with F1 scores below 0.10, well below the differential diagnosis ceiling of 0.20-0.27.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.07853 [cs.CL]
(or arXiv:2606.07853v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.07853 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 297 words
Related coverage
Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders
arXiv · Jul 21, 2026
Newer Models, Same Advantage
Hugging Face · Jul 16, 2026
Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
arXiv · Jun 3, 2026
MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts
arXiv · Oct 1, 2026