aiqbench analysis
This paper proposes a novel framework for evaluating large language models on subjective questions lacking ground truth by adapting validity concepts from stated-preference economics. By testing model outputs against theoretical economic predictions, the study demonstrates how to assess coherence and internal logic rather than factual correctness in complex decision-making scenarios. This approach offers a rigorous method for benchmarking AI systems in policy analysis and value-based reasoning where traditional accuracy metrics fail.
Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
arXiv:2610.10506v1 Announce Type: cross Abstract: Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
Abstract:Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); General Economics (econ.GN)
arXiv:2610.10506 [cs.AI]
(or arXiv:2610.10506v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.10506 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 302 words
Related coverage
LiveLaw Supreme Court Half-Yearly Digest: January - June 2026 [Part 2]
livelaw.in · Sep 13, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026