Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence

Abstract:Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date.

16 pages, 6 figures, 1 table. Evidence cutoff: 12 August 2026. Ancillary code and data included. Preprint; comments welcome

Artificial Intelligence (cs.AI)

arXiv:2608.14903 [cs.AI]

(or arXiv:2608.14903v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2608.14903 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

DATA_AND_SOURCE_AUDIT_v4.md

FIGURE_AND_PDF_QA_REPORT.txt

MEASUREMENT_DIRECTIONS_RESEARCH_v4.md

README_ANCILLARY.txt

model/analyse.py

model/analysis_output.txt

model/build_graph.py

model/build_model.py

model/data/benchmarks.csv

model/data/bridges.csv

model/data/compute_horizon.csv

model/data/criteria.csv

model/data/cutoff_release_check.csv

model/data/data_dictionary.csv

model/data/data_manifest.csv

model/data/data_quality_report.csv

model/data/dataset_inventory.csv

model/data/direction_source_links.csv

model/data/equating.csv

model/data/events.csv

model/data/graph.ttl

model/data/join_intersections.csv

model/data/labs.csv

model/data/literature_sources.csv

model/data/measurement_designs.csv

model/data/measurement_directions.csv

model/data/metr_th11.csv

model/data/model_observability.csv

model/data/models.csv

model/data/pairwise_overlap.csv

model/data/power.csv

model/data/relations.csv

model/data/source_audit.csv

model/data/source_verification_log.csv

model/data/sources.csv

model/directions_fig.py

model/equating.py

model/equating_fig.py

model/equating_output.txt

model/equating_results.json

model/figure_utils.py

model/finalize_data_inventory.py

model/graph_stats.txt

model/inventory_output.txt

model/lifecycle_fig.py

model/measurement_directions.py

model/measurement_directions_output.txt

model/method_fig.py

model/observability_fig.py

model/power_output.txt

model/research_audit.py

model/research_audit_output.txt

model/synthesis_fig.py

requirements.txt

(51 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.