analiza aiqbench
The paper proposes BRACE, a method for generating training data that tests whether AI verifiers check both a radiology report and its cited evidence, not just one of them. Its main finding is that verifiers trained without contradicted examples often accept AI-written statements that conflict with the report, while BRACE-trained verifiers reject most of them. Readers should note that the evaluation relies on public datasets, a food-allergen extension, and a small radiologist labeling study, so clinical readiness remains unproven.
Automated grounding verification of AI statements about chest radiographs
arXiv:2604.09537v2 Announce Type: replace-cross Abstract: A grounding verifier checks a statement about a patient's image against the radiology report and the cited evidence. The state-of-the-art methods for testing these verifiers remove or replace the cited evidence while the patient's report stays unchanged. To our knowledge, no existing method checks both the report and the cited evidence. We present Building Report And Citation Examples (BRACE), an automated framework that builds training examples from public radiology reports and Radiopaedia articles. We compared a lexical baseline and 42 verifiers with 0.1-284B parameters on 5 different public chest radiograph datasets. At a threshold accepting 0.90 of the supported test examples, verifiers trained with BRACE accepted a mean of only 0.239 of AI-written statements contradicted by the report. Verifiers trained without contradicted examples passed the state-of-the-art tests, yet accepted most of these statements (0.830 on average). With citations retrieved from held-out Radiopaedia articles, verifiers trained with BRACE accepted 0.169 of AI-written statements with an agreeing report and an unsupporting citation. By contrast, zero-shot large language models mostly checked the report and not the citation (they accepted 0.830 of these statements). We further assessed the framework manually, with two board-certified radiologists independently labeling a sample of test reports. The framework's labels agreed with each radiologist (k > 0.910) almost as closely as the radiologists agreed with each other (k = 0.952). Moreover, instantiating the framework for food allergens in recipes, we showed that it extends beyond clinical applications. A verifier built the same way accepted 0.493 of the statements contradicted by the recipe, whereas a lexical baseline accepted 0.898. Using both the report and the cited evidence, verifiers trained with BRACE could check AI statements before clinicians read them.
Abstract:A grounding verifier checks a statement about a patient's image against the radiology report and the cited evidence. The state-of-the-art methods for testing these verifiers remove or replace the cited evidence while the patient's report stays unchanged. To our knowledge, no existing method checks both the report and the cited evidence. We present Building Report And Citation Examples (BRACE), an automated framework that builds training examples from public radiology reports and Radiopaedia articles. We compared a lexical baseline and 42 verifiers with 0.1-284B parameters on 5 different public chest radiograph datasets. At a threshold accepting 0.90 of the supported test examples, verifiers trained with BRACE accepted a mean of only 0.239 of AI-written statements contradicted by the report. Verifiers trained without contradicted examples passed the state-of-the-art tests, yet accepted most of these statements (0.830 on average). With citations retrieved from held-out Radiopaedia articles, verifiers trained with BRACE accepted 0.169 of AI-written statements with an agreeing report and an unsupporting citation. By contrast, zero-shot large language models mostly checked the report and not the citation (they accepted 0.830 of these statements). We further assessed the framework manually, with two board-certified radiologists independently labeling a sample of test reports. The framework's labels agreed with each radiologist (k > 0.910) almost as closely as the radiologists agreed with each other (k = 0.952). Moreover, instantiating the framework for food allergens in recipes, we showed that it extends beyond clinical applications. A verifier built the same way accepted 0.493 of the statements contradicted by the recipe, whereas a lexical baseline accepted 0.898. Using both the report and the cited evidence, verifiers trained with BRACE could check AI statements before clinicians read them.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG)
arXiv:2604.09537 [cs.CL]
(or arXiv:2604.09537v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2604.09537 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 380 cuvinte
Subiecte conexe
XMedFusion: A Knowledge-Guided Multimodal Perception and Reasoning Framework for Autonomous Medical Systems
arXiv · 16 iun. 2026
Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
arXiv · 15 oct. 2026
CheXPoT: Post-Hoc Visual Verification and Correction of VLM-Generated Radiology Reports
arXiv · 3 nov. 2026