aiqbench analysis
AgentSLR introduces a rigorous evaluation harness for systematic literature reviews in epidemiology, revealing that no single frontier model dominates across all tasks and highlighting structured data extraction as a critical bottleneck with sub-0.67 F1 scores. The study underscores significant cost disparities of up to 96x between models and confirms that current LLMs lack the reliability required for unsupervised deployment in high-stakes public policy contexts. This work shifts the focus from aggregate benchmarks to granular failure analysis, emphasizing the need for specialized sub-task evaluation in scientific knowledge synthesis.
Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews
arXiv:2603.22327v2 Announce Type: replace-cross Abstract: Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.
Authors:Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo, Lingyi Yang, Łukasz Borchmann, Piotr BŁaszczyk, Christian Morgenstern, Ruth McCabe, Sangeeta Bhatia, Philip H. Torr, Jakob Foerster, Scott A. Hale, Thomas Rawson, Anne Cori, Elizaveta Semenova, Adam Mahdi
Abstract:Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL)
arXiv:2603.22327 [cs.IR]
(or arXiv:2603.22327v2 [cs.IR] for this version)
Full text under the publisher's open licence · 220 words
Related coverage
Large language model-generated patterns for epidemiological modeling: a multi-benchmark evaluation framework
arXiv · Oct 15, 2026
Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests
arXiv · Oct 22, 2026
Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records
arXiv · Jun 9, 2026
EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning
arXiv · Sep 1, 2026