analiza aiqbench
This paper proposes a structured rubric to evaluate clinical reasoning in large language models, integrating medical education frameworks with existing LLM benchmarks and general reasoning taxonomies. By explicitly flagging safety-critical errors and grounding factuality checks in clinical contexts, it aims to make evaluation decisions transparent before empirical validation occurs. The work addresses the critical need for rigorous assessment standards as AI moves toward specialized healthcare applications.
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
arXiv:2609.37788v1 Announce Type: cross Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench, HealthBench, TIMER-Bench, DR.BENCH, PrIME-LLM and PatientSafeBench); and general LLM reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.
Abstract:Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench, HealthBench, TIMER-Bench, this http URL, PrIME-LLM and PatientSafeBench); and general LLM reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.37788 [cs.CL]
(or arXiv:2609.37788v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.37788 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 264 cuvinte
Subiecte conexe
Always ask your agent to be honest
LessWrong · 16 sept. 2026
Models Can't Remember Their Training. Neither Can You.
LessWrong · 19 iul. 2026
Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training
arXiv · 12 sept. 2026
ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail
arXiv · 15 oct. 2026