aiqbench analysis
This paper tackles the critical reproducibility crisis in social science research by proposing a probabilistic standard and stacked ensemble methods to mitigate LLM output variance. It offers a technical framework for diagnosing model errors and restoring statistical validity without sacrificing precision, directly addressing methodological challenges in using AI for empirical analysis.
Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models
arXiv:2611.06639v1 Announce Type: new Abstract: This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.
Abstract:This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.
Chapter 5 of "LLMs in Political and Social Science: Principles and Applications of Transformer Models," Oxford University Press, forthcoming 2027. Data and code: this https URL
Computation and Language (cs.CL); Computers and Society (cs.CY)
arXiv:2611.06639 [cs.CL]
(or arXiv:2611.06639v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.06639 arXiv-issued DOI via DataCite (pending registration)
From: Joan C. Timoneda [view email] [v1] Wed, 4 Nov 2026 15:20:03 UTC (230 KB)
Full text under the publisher's open licence · 202 words
Related coverage
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · Oct 8, 2026
Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
arXiv · Jun 4, 2026
Anthropomimetic Soft Robotic Forearm with Independently Articulated Carpal Bones Enabling Human-Like Adaptive Stiffness Modulability
arXiv · Sep 25, 2026
Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5
arXiv · Aug 18, 2026