What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals

Title:What Does an Evaluation License? A Commit-Bound Census of Claim-Relative Inference in Inspect Evals

Abstract:Evaluation artifacts specify a forward computation: a task, scorer, and reported metric. They do not necessarily license the claim attached to that metric because the historical evidence and alternative semantics needed to replay it may be unbound. We formalize this missing claim-replay layer through a frozen substrate D, a grounded family F, a claim query q, and the resulting identified set. We then census all 124 mechanically eligible Inspect Evals units at a pinned commit. Every unit receives a terminal disposition; 110 stop before deterministic inference because required historical evidence or semantic grounding is unavailable. Where execution closes, exact values, winners, complete orders, and pairwise relations separate by claim resolution and by primary versus review family. The audit therefore returns typed stops, instability witnesses, and stable substructure rather than forcing one evaluator meaning or one robust/not-robust label.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI)

arXiv:2608.19269 [cs.SE]

(or arXiv:2608.19269v3 [cs.SE] for this version)

https://doi.org/10.48550/arXiv.2608.19269 Focus to learn more arXiv-issued DOI via DataCite

census/CENSUS_SUMMARY.json

census/TERMINAL_RECORDS.json

contract/CONTRACT.md

contract/records/agentdojo/audit.normalized.json

contract/records/agentdojo/report.json

contract/records/automlbenchmark/audit.normalized.json

contract/records/automlbenchmark/report.json

contract/records/beir-scifact/audit.normalized.json

contract/records/beir-scifact/report.json

evidence/ANALYSIS_ROLES.json

evidence/EXPLORATORY_SOURCE_AUTHORIZATION.json

recomputation/INDEPENDENT_RECOMPUTATION.json

recomputation/source_tables/ANIMA.md

recomputation/source_tables/BFCL.md

recomputation/source_tables/MORU.md

reviewer/MANUSCRIPT_BINDING.json

reviewer/RESULTS.md

verification/verify_supplement.py

(16 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.