aiqbench analysis
The paper applies LLMs to a practical problem: scoring attending surgeons' verbal feedback during live operations. Its method uses repeated sampling and consolidation to propose interpretable criteria such as encouragement, urgency, and clarity, then applies them via LLM-as-a-judge. The reported gain over content-based baselines on 4.2k instances is promising, but readers should check how well the LLM judge agrees with expert raters and whether the results generalize beyond this dataset.
A Multi-Instance LLM Framework for Rating the Quality of Surgical Feedback
arXiv:2605.25440v2 Announce Type: replace-cross Abstract: Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies assessed feedback content relying on extensive manual annotation by expert human raters and focused on developing broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced aspects. We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training. Our method uses multi-instance prompting via repeated independent sampling of candidate criteria followed by consolidation, and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria (e.g., Encouragement, Urgency, Clarity). These criteria are then used to automatically score live surgical feedback via an LLM-as-a-judge approach. Evaluation on 4.2k trainer feedback instances demonstrates that our AI-discovered criteria outperform prior content-based frameworks in predicting feedback effectiveness, including observed trainee behavioral adjustments and trainer approval. This work advances scalable, human-aligned assessment of communication quality in the operating room and provides a foundation for improving surgical teaching practices.
Abstract:Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies assessed feedback content relying on extensive manual annotation by expert human raters and focused on developing broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced aspects. We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training. Our method uses multi-instance prompting via repeated independent sampling of candidate criteria followed by consolidation, and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria (e.g., Encouragement, Urgency, Clarity). These criteria are then used to automatically score live surgical feedback via an LLM-as-a-judge approach. Evaluation on 4.2k trainer feedback instances demonstrates that our AI-discovered criteria outperform prior content-based frameworks in predicting feedback effectiveness, including observed trainee behavioral adjustments and trainer approval. This work advances scalable, human-aligned assessment of communication quality in the operating room and provides a foundation for improving surgical teaching practices.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
I.2.7; J.3; K.3.1
arXiv:2605.25440 [cs.CL]
(or arXiv:2605.25440v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.25440 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 309 words