Towards AI epidemiology: a measurement standardisation framework for prospective risk detection

arXiv:2512.15783v4 Announce Type: replace Abstract: This paper proposes a measurement standardisation framework that compresses expert-AI interactions into structured, comparable fields for prospective risk detection in deployed AI systems, without access to model internals. This concept paper defines the framework's scope, semantically and statistically, and specifies a protocol for its empirical testing. The population-level claims it is designed to support therefore belong to a staged research programme rather than to results claimed here. Measurement standardisation underpins three claims. The first is a reliability claim: under bounded conditions, large language models can produce reliable, standardised assessments of the evidential and policy alignment of expert-AI interactions. The second is a governance claim: alignment scores give experts an immediate signal during deployment and give institutions a basis for monitoring alignment patterns across mission types, models, and domains. The third is an outcome validation claim: once measurement standardisation is established, aggregate alignment scores could be used to study associations with downstream outcomes in regulated professional settings. This introduces the possibility of an "AI epidemiology", a form of risk detection based on correlated variables instead of mechanistic analysis, inspired by epidemiological reasoning. A minimal application of the protocol to a published expert-AI corpus shows that the judge reproduces its policy and evidential alignment scores across two runs under the specified conditions. Judge reliability at scale remains to be validated in future work. The paper sets out a defined grammar of eight interaction fields, together with a statistical protocol based on paired bootstrap inference, DeLong's test for paired AUCs as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm-Bonferroni correction.

Abstract:This paper proposes a measurement standardisation framework that compresses expert-AI interactions into structured, comparable fields for prospective risk detection in deployed AI systems, without access to model internals. This concept paper defines the framework's scope, semantically and statistically, and specifies a protocol for its empirical testing. The population-level claims it is designed to support therefore belong to a staged research programme rather than to results claimed here. Measurement standardisation underpins three claims. The first is a reliability claim: under bounded conditions, large language models can produce reliable, standardised assessments of the evidential and policy alignment of expert-AI interactions. The second is a governance claim: alignment scores give experts an immediate signal during deployment and give institutions a basis for monitoring alignment patterns across mission types, models, and domains. The third is an outcome validation claim: once measurement standardisation is established, aggregate alignment scores could be used to study associations with downstream outcomes in regulated professional settings. This introduces the possibility of an "AI epidemiology", a form of risk detection based on correlated variables instead of mechanistic analysis, inspired by epidemiological reasoning. A minimal application of the protocol to a published expert-AI corpus shows that the judge reproduces its policy and evidential alignment scores across two runs under the specified conditions. Judge reliability at scale remains to be validated in future work. The paper sets out a defined grammar of eight interaction fields, together with a statistical protocol based on paired bootstrap inference, DeLong's test for paired AUCs as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm-Bonferroni correction.

38 pages, 3 figures, 4 tables. Accepted for publication in AI & Society

Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

arXiv:2512.15783 [cs.AI]

(or arXiv:2512.15783v4 [cs.AI] for this version)

aiqbench analysis

Towards AI epidemiology: a measurement standardisation framework for prospective risk detection

Related coverage