aiqbench analysis
The benchmark tests 28 LLMs on 142 Australian clinical-policy questions with exact-match scoring. Most failures (86%) disappeared when the governing clause was supplied, which suggests the problem is mainly missing or outdated local knowledge rather than an inability to apply rules. The answer keys came from one GP and are unreviewed, and the results apply to base models, not clinical products, so they show a gap to address, not a clearance for clinical use.
AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy
arXiv:2610.31993v1 Announce Type: cross Abstract: AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.
Abstract:AI models already inform clinical decisions and draft notes in Australian clinical software, usually without the practitioner being told which model is used. Existing medical benchmarks are written for other jurisdictions. We wrote 142 questions from Australian guidelines and schedules across five domains (PBS authority, MBS billing, Ahpra obligations, triage red flags, immunisation and screening), each with one verifiable answer scored by exact match. Each question ran blind and with the governing clause supplied, which separates failure to recall the current rule from failure to apply it. We tested 28 models between July and October 2026. Blind accuracy ranged from 51.5% to 92.7%, and the highest-scoring models could not be separated. Of 791 failed model-question pairs, 684 (86%) were wrong blind and right with the clause. Confident-error rates ranged from 6.6% to 47.8%. On five immunisation questions where the guideline gives no directive, 82% of answers stated one. On 20 Aboriginal and Torres Strait Islander health questions, models did no worse overall, but on a few the wrong answers gave the rule for the general population, an older rule or another country's rule. We propose treating the model as an ingredient of a clinical product, with a minimum standard on local rules that sits beside product evaluation. The keys are one general practitioner's and are not yet independently reviewed. The benchmark measures base models, not clinical products, and does not license any model for clinical use.
19 pages, 3 figures. Technical report, version 1.4.1 of the benchmark. Not peer reviewed. Answer keys not yet independently reviewed. Results and methods: this https URL
Computers and Society (cs.CY); Computation and Language (cs.CL)
arXiv:2610.31993 [cs.CY]
(or arXiv:2610.31993v1 [cs.CY] for this version)
https://doi.org/10.48550/arXiv.2610.31993 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 285 words