analiza aiqbench
This research identifies a critical "preformulation gap" in medical LLMs, where models often provide premature self-care advice or fail to structure handoffs when patients present vague initial concerns rather than clear clinical problems. The study demonstrates that current evaluation metrics focusing solely on diagnostic accuracy are insufficient because they ignore the crucial early-stage interaction dynamics essential for real-world consultation safety and efficacy. For AI developers in healthcare, this underscores the urgent need to shift testing paradigms toward observable first-contact behaviors before a patient's problem is fully defined.
LLMs for Medical Consultation Are Evaluated Too Late: The Preformulation Gap
arXiv:2608.17330v1 Announce Type: cross Abstract: Large language models for medical consultation are often evaluated after a clinical problem has already been made clear, although real consultations may begin with a vague, minimized, or misframed concern. We evaluated three API models across four physician-authored, multi-turn vignettes under baseline and entry-to-care instruction conditions, yielding 24 fixed-script transcripts; two cases also used adaptive standardized-patient simulation, yielding 12 transcripts. Self-care or home-management advice before any patient answer appeared in 9 of 12 baseline case-model cells and 0 of 12 instruction cells, while structured handoff summaries appeared in 0 of 12 and 10 of 12 cells, respectively. The instruction changed sequencing and documentation, although it did not reliably ensure elicitation of decisive facts. The preformulation gap should therefore be evaluated directly through observable first-contact behavior rather than inferred from diagnostic accuracy or final-answer quality.
Abstract:Large language models for medical consultation are often evaluated after a clinical problem has already been made clear, although real consultations may begin with a vague, minimized, or misframed concern. We evaluated three API models across four physician-authored, multi-turn vignettes under baseline and entry-to-care instruction conditions, yielding 24 fixed-script transcripts; two cases also used adaptive standardized-patient simulation, yielding 12 transcripts. Self-care or home-management advice before any patient answer appeared in 9 of 12 baseline case-model cells and 0 of 12 instruction cells, while structured handoff summaries appeared in 0 of 12 and 10 of 12 cells, respectively. The instruction changed sequencing and documentation, although it did not reliably ensure elicitation of decisive facts. The preformulation gap should therefore be evaluated directly through observable first-contact behavior rather than inferred from diagnostic accuracy or final-answer quality.
17 pages, 3 tables. Code, cases, prompts, complete transcripts, and results: this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2608.17330 [cs.AI]
(or arXiv:2608.17330v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.17330 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 250 cuvinte
Subiecte conexe
Links #5: 2026/07
LessWrong · 31 iul. 2026
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
arXiv · 17 iun. 2026
Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation
arXiv · 11 sept. 2026
Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations
arXiv · 2 oct. 2026