aiqbench analysis
This research investigates the specific failure modes of Large Language Models in dynamic clinical diagnosis, shifting focus from static knowledge possession to the mechanisms of evidence acquisition and utilization over time. By analyzing trajectory dynamics and information entropy, the study identifies critical capability gaps where models struggle to integrate new diagnostic evidence effectively. These findings are essential for advancing reliable AI-assisted decision-making systems in high-stakes medical environments.
From Hindsight to Foresight: Dissecting Clinical Evidence Acquisition in Large Language Models
arXiv:2610.18605v1 Announce Type: new Abstract: Large language models (LLMs) often diagnose full sight, static clinical cases well yet struggle in foresight, dynamic diagnosis. Prior work has improved this limitation by quantifying the information models acquire and by developing stronger follow-up strategies to improve diagnostic accuracy. In contrast, we introduce a distinctive hindsight to systematically dissect why LLMs fail to acquire diagnostically critical evidence, tracing the failure from evidence identification, utilization and downstream diagnostic updating through not only diagnostic accuracy, but also evidence coverage, trajectory dynamics, and elicited probability and information entropy. Starting experiments from the breakdown at the level of capabilities, to the diagnosis of the pre-event state before the evidence arrives, and to the post-event state transition after the evidence arrives, our research has revealed the profound limitations and internal causes existing in the dynamic diagnostic process. At the same time, we have also conducted sufficient verification and generalization experiments to prove that these limitations are widespread. Together, these findings shift the central challenge from whether LLMs possess relevant clinical knowledge to whether they can identify, acquire, and effectively use the right evidence under clinical diagnosis and conduct a more detailed mechanism analysis to identify the exact location of the capability gap.
Abstract:Large language models (LLMs) often diagnose full sight, static clinical cases well yet struggle in foresight, dynamic diagnosis. Prior work has improved this limitation by quantifying the information models acquire and by developing stronger follow-up strategies to improve diagnostic accuracy. In contrast, we introduce a distinctive hindsight to systematically dissect why LLMs fail to acquire diagnostically critical evidence, tracing the failure from evidence identification, utilization and downstream diagnostic updating through not only diagnostic accuracy, but also evidence coverage, trajectory dynamics, and elicited probability and information entropy. Starting experiments from the breakdown at the level of capabilities, to the diagnosis of the pre-event state before the evidence arrives, and to the post-event state transition after the evidence arrives, our research has revealed the profound limitations and internal causes existing in the dynamic diagnostic process. At the same time, we have also conducted sufficient verification and generalization experiments to prove that these limitations are widespread. Together, these findings shift the central challenge from whether LLMs possess relevant clinical knowledge to whether they can identify, acquire, and effectively use the right evidence under clinical diagnosis and conduct a more detailed mechanism analysis to identify the exact location of the capability gap.
Computation and Language (cs.CL)
arXiv:2610.18605 [cs.CL]
(or arXiv:2610.18605v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.18605 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 293 words
Related coverage
Inside the suddenly explosive world of AI safety
The Verge · Sep 17, 2026
From Uncertainty to Clinical Risk: Severity-Aware Conformal Planning for Interactive Medical Diagnosis
arXiv · Sep 1, 2026
EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents
arXiv · Aug 27, 2026
MedCTA: A Benchmark for Clinical Tool Agents
arXiv · Jun 11, 2026