aiqbench analysis
This paper addresses a critical epistemic flaw in AI evaluation where valid individual benchmark results fail to support chained projections due to shifting system conditions or data dependencies. By introducing a "non-composition principle," the authors provide an audit framework for diagnosing unsupported logical jumps between isolated studies and real-world deployment claims. This work is essential for researchers aiming to build robust, transferable evidence chains rather than relying on fragile extrapolations from single-point benchmarks.
When benchmark inferences do not compose: Projectibility in AI evaluation
arXiv:2607.26159v1 Announce Type: new Abstract: An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.
Abstract:An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.
35 pages, 2 figures, 5 tables. Substantially rewritten and retitled; supersedes arXiv:2510.15236, whose homeostatic property-cluster account and proposed centrality-prior and cluster-stability measures are withdrawn. The argument, apparatus, and empirical companion are new. Code and empirical companion: this https URL
Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG)
arXiv:2607.26159 [cs.AI]
(or arXiv:2607.26159v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.26159 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 267 words
Related coverage
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
News
Quantum Computing Report · Jun 7, 2026
Enabling Language-specific Reasoning in Multilingual Models with Reinforcement Learning
AMD · Jul 24, 2026