aiqbench analysis
This research demonstrates a practical application of multi-agent large language models augmented by human oversight to significantly improve the detection and classification of immune-related adverse events in clinical data. By combining retrieval-augmented generation with expert review, the framework achieves higher accuracy and faster processing times than manual methods alone, offering a scalable solution for pharmacovigilance tasks.
Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events
arXiv:2607.20428v1 Announce Type: new Abstract: This study evaluated a retrieval-augmented, multi-agent large language model (LLM)-driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with unassisted manual review, the LLM-assisted workflow improved accuracy (F1 = 0.88 vs 0.77), inter-rater agreement measured by Cohen's kappa (kappa = 0.82 vs 0.50), and reduced average review time by approximately half. This framework pilots how LLMs can be applied to identify immune-related toxicities across organ systems and, more broadly, enable accurate, scalable, and transparent adverse event data extraction.
Authors:Charles Lu, Olivia Burke, Debby Cheng, Adam Kashlan, Caitlyn Duffy, Zeyun Lu, Lirit Fuksman, Jin Ning Tian, Andrew Sedlack, Priya Katyal, Eudora Lee, Ralina Karagenova, Chuck Lin, Kun-Hsing Yu, Nicole LeBoeuf, Alexander Gusev, Yevgeniy R. Semenov
Abstract:This study evaluated a retrieval-augmented, multi-agent large language model (LLM)-driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with unassisted manual review, the LLM-assisted workflow improved accuracy (F1 = 0.88 vs 0.77), inter-rater agreement measured by Cohen's kappa (kappa = 0.82 vs 0.50), and reduced average review time by approximately half. This framework pilots how LLMs can be applied to identify immune-related toxicities across organ systems and, more broadly, enable accurate, scalable, and transparent adverse event data extraction.
Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Multiagent Systems (cs.MA)
arXiv:2607.20428 [cs.CL]
(or arXiv:2607.20428v1 [cs.CL] for this version)
Full text under the publisher's open licence · 137 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Reducing medical claims review time with AI on AWS: The EXL Medical IDP solution
AWS · Sep 21, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
Six Principles of Voice Design: Designing for the Ear and Encoding It into Agent Script
Salesforce Blog · Oct 8, 2026