aiqbench analysis
This forensic audit exposes critical reproducibility failures in a radiology vision-language model benchmark, revealing that original performance claims were invalidated by data leakage, incorrect DICOM rendering, and truncated outputs. The study fundamentally challenges the reliability of current medical-imaging AI benchmarks where artifacts like prompt bindings and dataset splits are assumed correct rather than empirically verified. By withdrawing prior clinical assertions and proposing machine-verifiable controls for cohort integrity and model identity, this work sets a necessary standard for rigorous validation in high-stakes healthcare AI applications.
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
arXiv:2607.25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases. Agreement across these artifacts is usually assumed rather than tested. We performed a retrospective forensic reproducibility audit of a preserved chest-radiograph vision-language model (VLM) pilot; no model was called again and no image or report was newly annotated. We traced prompt bindings, DICOM metadata, output completeness, label extraction, matched analyses, and release propagation. Of 300 planned model-prompt calls, 297 yielded nonempty reports. Sixty Claude calls labeled A/B were executed with the same C prompt. The 30 studies represented 28 patients. Four MONOCHROME1 images were rendered without required polarity inversion, dataset split membership was not retained, and the unvalidated extractor truncated five reports to 4000 characters. Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29. Of 45 McNemar comparisons, 27 had unadjusted p < 0.05 and 20 remained below 0.05 after Holm adjustment. These values describe only the archived automated-label matrix; they do not recover the intended prompt comparison or establish clinical performance. We withdraw the original performance, ranking, prompt-effect, and clinical claims and specify machine-verifiable controls for cohort, DICOM rendering, prompt and model identity, call status, annotation provenance, keyed analysis, and derived artifacts.
Abstract:Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases. Agreement across these artifacts is usually assumed rather than tested. We performed a retrospective forensic reproducibility audit of a preserved chest-radiograph vision-language model (VLM) pilot; no model was called again and no image or report was newly annotated. We traced prompt bindings, DICOM metadata, output completeness, label extraction, matched analyses, and release propagation. Of 300 planned model-prompt calls, 297 yielded nonempty reports. Sixty Claude calls labeled A/B were executed with the same C prompt. The 30 studies represented 28 patients. Four MONOCHROME1 images were rendered without required polarity inversion, dataset split membership was not retained, and the unvalidated extractor truncated five reports to 4000 characters. Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29. Of 45 McNemar comparisons, 27 had unadjusted p < 0.05 and 20 remained below 0.05 after Holm adjustment. These values describe only the archived automated-label matrix; they do not recover the intended prompt comparison or establish clinical performance. We withdraw the original performance, ranking, prompt-effect, and clinical claims and specify machine-verifiable controls for cohort, DICOM rendering, prompt and model identity, call status, annotation provenance, keyed analysis, and derived artifacts.
25 pages, 5 figures, 7 tables. Retrospective artifact audit; no new model calls or annotations. Corrective audit and archival reconstruction: this https URL
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
I.2.10; D.2.4; D.2.5
arXiv:2607.25589 [cs.CV]
(or arXiv:2607.25589v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2607.25589 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 259 words
Related coverage
Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning
NVIDIA · Sep 23, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
[AINews] High Taste Testers Ep. 1
Latent Space · Oct 17, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026