How the leaderboards work

1. Reading a leaderboard row

The weighted-blend view ("build your own view") shows one row per model with five columns beyond the model name. Each carries a specific claim; each has a specific confidence shape.

Composite

A 0–100 percentile-blend across the selected axes. The reader's own weight vector determines how much each axis contributes. Never a blend of raw scores — Elo and pass-rate don't share a unit.

Coverage

How many of the selected axes this model actually appears on. Rows are ordered on the coverage-shrunk composite (not the raw one), so a strong number from 2 boards no longer outranks a broad record from 15. A model needs about 3 axes of evidence before its own record outweighs the neutral prior.

[low – high]

The composite carried through source-published confidence intervals. When every axis reports a CI, the interval is honest and can be narrower or wider than the point estimate suggests. When no axis reports one, the interval collapses to the point value — nothing invented.

Divergence

Max minus min percentile across the axes this model actually appears on. High divergence surfaces asymmetric performers — great on one axis, weak on another — whose single composite number hides the story.

Rank swing

How much a model's rank moves across the five built-in template weightings (all-rounder / coding-focused / conversation-first / reasoning / agentic). Low swing means the composite is robust to the weight vector; high swing means the specific rank the reader sees depends heavily on which template drove it.

The default sort is by coverage-shrunk composite, descending. Every row is a claim from live data; hovering any composite cell reveals the raw value, the CI range, and the shrinkage that was applied.

2. Data sources

Every axis on this site is ingested from one publicly published upstream leaderboard — arena-style Elo boards, measured benchmarks, agentic harnesses and hallucination rates alike. Each board keeps its own publisher, cadence, licence and attribution; nothing is blended at ingestion. Each source's dataset URL, licence and full attribution string is on the Sources page.

The list below is generated from the same board registry the Sources page renders — 22 publishers, 53 boards today — so the two pages cannot disagree about what is ingested.

• ARC Prize — 1 board: ARC-AGI-2
• arena.ai (LMArena) — 10 boards: Arena · Agent, Arena · Document, Arena · Factuality, Arena · Image Edit, Arena · Search, Arena · Text, Arena · Text-to-Image, Arena · Text-to-Video, Arena · Vision, Arena · WebDev
• Artificial Analysis — 4 boards: IFBench (Artificial Analysis), LiveCodeBench (Artificial Analysis), MMLU-Pro (Artificial Analysis), Output speed, tokens/s (Artificial Analysis)
• awesome-quantum-computing-experiments (François-Marie Le Régent) — 4 boards: Entangled-state and two-qubit gate error, Physical qubit coherence — T1, Physical qubit count records, Quantum error correction experiments — code distance
• Berkeley Function Calling Leaderboard (Gorilla, UC Berkeley) — 1 board: BFCL v4 — function calling
• Epoch AI — 5 boards: FrontierMath, GPQA Diamond, OTIS Mock AIME, SimpleQA Verified, SWE-bench Verified
• Holistic Agent Leaderboard (Princeton) — 2 boards: GAIA (HAL), USACO (HAL)
• IBM Quantum Platform API — 1 board: IBM Quantum fleet — live systems
• IonQ Quantum Cloud API — 1 board: IonQ fleet — live backends
• LiveBench — 1 board: LiveBench (overall)
• metriq-data (Unitary Foundation) · QED-C QFT circuit — 1 board: QED-C circuits on hardware (metriq-data)
• Metriq (Unitary Foundation) — metriq-gym runs — 7 boards: BSEQ — largest connected qubit set (Metriq), CLOPS — circuit layer operations per second (Metriq), EPLG — error per layered gate (Metriq), Linear Ramp QAOA at 20 qubits (Metriq), Mirror circuits polarization, 16 qubits × 32 layers (Metriq), QML Kernel accuracy at 20 qubits (Metriq), WIT witness expectation at 7 qubits (Metriq)
• Quantum Advantage Tracker — 3 boards: Quantum Advantage — Classically-Verifiable Problems, Quantum Advantage — Observable Estimations, Quantum Advantage — Variational Problems
• Scale AI / CAIS — 1 board: Humanity's Last Exam
• SimpleBench — 1 board: SimpleBench
• SWE-bench Verified leaderboard — 1 board: SWE-bench Verified (live board)
• Terminal-Bench 4.0 (Harbor / Laude Institute) — 1 board: Terminal-Bench 4.0
• Terminal-Bench (Laude Institute) — 1 board: Terminal-Bench
• TIGER-Lab (University of Waterloo) — 1 board: MMLU-Pro (TIGER-Lab)
• Vectara — 1 board: Vectara HHEM Hallucination Leaderboard
• Wikipedia — 2 boards: Quantum processors (Wikipedia), Quantum volume records (Wikipedia)
• τ²-bench via the Exgentic Open Agent Leaderboard — 3 boards: τ²-bench Airline, τ²-bench Retail, τ²-bench Telecom

The Sources page also lists the modifications we apply to every ingested board — subset selection, schema normalization, rank recomputation on our normalized identifier — so the reader can see exactly what changed between the upstream dataset and the number rendered here.

Sources & attribution →

3. Composite math

The composite is a weighted average of per-axis percentiles. Never of raw scores. The three steps below fully define it.

Step 1 — Per-axis percentile ranking. On each axis, models are ranked 0–100 within that axis's own population. A model that scores near the top of a small-population axis gets a high percentile; the same raw score would be lower on a large-population axis. This makes percentiles unitless and comparable across boards in a way raw Elo/percent scores are not.

Step 2 — Weighted blend. The reader (or a built-in template) supplies a weight for each selected axis, between 0 and 10. A model's composite is the weight-normalized sum of its percentiles across axes on which it has data. Axes it doesn't appear on are excluded from that model's blend rather than treated as zero — a missing measurement is not a bad measurement.

Step 3 — Coverage-shrinkage. A raw mean over few axes is not comparable to a raw mean over many. Because one lucky board can produce a 99th-percentile mean with no broad support, every composite is shrunk toward the neutral prior in proportion to how little evidence backs it.

The prior is 3 axes worth of neutral 50th percentile. The formula is the standard small-sample shrinkage:

adjusted = (n · mean + 3 · 50) / (n + 3)

Coverage counts ("how many of the axes selected for this view a given model actually appears on") is surfaced on every row. A one-axis composite is technically defined but visibly low-confidence, and the raw pre-shrinkage value is kept visible so the reader can see what the shrinkage did.

4. Confidence intervals

Some upstream sources publish a per-score confidence interval — LMArena's rating_lower / rating_upper, Epoch's stderr expanded to a ±1σ band, and so on. Some don't (HHEM, MMLU-Pro). The composite honestly propagates whatever the sources publish and does not invent uncertainty they didn't declare.

Propagation. Each axis's raw CI is percentile-ranked in the same population as its score, producing a percentileLow and percentileHigh. The composite's low and high bounds are the same weight-normalized sum applied to those percentile-CIs. When every axis publishes a CI, low ≤ composite ≤ high; when none do, low = composite = high.

A wide-CI axis carries its width through the composite; a tight-CI axis carries tight bounds. The reader hovering a composite cell sees both the raw point value and the honest [low, high] interval — a wide interval reads as less confident than a narrow one, exactly as it should.

5. Divergence

Divergence for a row is the percentile range across the axes it actually appears on: the highest-percentile axis minus the lowest-percentile axis, expressed in percentile points. A model that lands at the 95th on one axis and the 30th on another has a divergence of 65. A model that lands at 62 on every axis has a divergence of 0.

The /leaderboards/insights page surfaces at most 5 divergence-worthy rows across the whole population; the threshold for eligibility is 40 percentile points of spread. Below that, the pattern is "models are uneven in the small" — not a story worth naming. The main-page divergence column shows every row's value; the insights page only names the extremes.

6. Weight sensitivity

A composite carries one specific weight vector. A reader looking at rank #3 doesn't automatically know whether that model is #3 under most reasonable weightings or #3 only under this one. Rank swing measures that: for each model, we compute its rank under each of the five built-in templates and report the min, the max, and the spread. Discrete templates (not random weights) because those are the actual weight vectors a reader is plausibly considering — a pathological weight vector nobody would pick shouldn't skew the volatility signal.

• Stable: rank spread ≤ 5 across templates. The specific rank the reader sees is robust — this model lands in about the same place regardless of which template drives the composite.
• Moderate: rank spread between 5 and 20. The rank depends somewhat on the template. The composite reads honestly but the specific rank is not template-independent.
• Volatile: rank spread above 20. The composite is heavily template-dependent — do not read the visible rank as a robust fact.
• Single-template: the model only appears in one template's blend. Sensitivity requires at least two ranks to compare, so no swing figure is emitted; the row gets a distinct badge rather than a fake number.

The insights page names at most 5 volatile rows; eligibility for the list is spread ≥ 20 — matching the volatile-bucket boundary.

Sensitivity across random weight vectors is a cleaner mathematical concept but pragmatically misleading — averaging a template no reader would pick into the spread over-inflates the volatility signal. The templates are the honest discrete choice set.

7. Axis clusters

Some axes measure similar skills (multiple coding boards, multiple reasoning boards). We detect this from the data rather than assert it editorially. For every pair of axes with at least 20 models in common, we compute the Pearson correlation of their percentile vectors on those shared models. Two axes share a cluster iff their pairwise r ≥ 0.85.

Cluster membership is the transitive closure of those edges — connected components on the correlation graph. Deliberately not k-means or hierarchical: connected components are auditable ("these two axes correlated above threshold, walk the edges"), and they let the data decide the cluster count instead of requiring one up-front.

Correlation is not semantic identity. Two axes may correlate because model scale drives both — that fact does not license the reader to treat them as measuring the same skill. Editorial cluster labels ("The Coding Cluster") are named as such and only applied to pure-family clusters; a mixed-family cluster gets no descriptor.

Two failure modes are reported separately, not merged. An axis with adequate overlap but no cluster-worthy correlation is a singleton (measured, independent). An axis with zero adequately-covered pairs is unclusterable (not enough data to say anything). Distinguishing them lets a reader decide whether the silence on an axis is a real result or a data gap.

8. Arrival timeline

The arrival-timeline view plots one point per model at its release date, on one Epoch benchmark, with a running-max frontier line above showing the best-known score at any point in time. This is a real trend — as better models arrive, the frontier rises.

Currently eligible boards: GPQA Diamond, OTIS Mock AIME, FrontierMath, SimpleQA Verified, SWE-bench Verified, ARC-AGI-2, Terminal-Bench, Humanity's Last Exam, SimpleBench. Epoch only, deliberately. LMArena scores are Elo (not fixed capability metrics, and they shift with the vote population), so plotting one Elo per model on release date would invite reading a specific model's Elo as "what it was when it launched," which is not true.

The x-axis is the model's own release date, resolved by our curated → name-suffix → HuggingFace createdAt → LMArena first-seen chain (98.6% coverage globally, 79–95% per eligible board). Deliberately NOT Epoch's own publishedAt field — for boards that use "Started at" in the CSV, that column is the evaluation timestamp (when Epoch ran the eval), which for older models routinely lands months after the model's actual release.

Sparse-data guard. A scatter with 3 dated points isn't a trend, it's an insinuation. Boards with fewer than 10 dated points render a "not enough data" state instead of the chart. The number is a judgement, chosen small enough to include narrow benchmarks like SWE-bench Verified without letting a fresh board show a straight line through 2 points.

What this is NOT: a model's score over time. A fixed model's benchmark score is fixed — Claude 3.5 Sonnet's GPQA score today is the same number Anthropic reported at launch. Charting a single model's history as if it were a time series would misrepresent the phenomenon and mislead the reader.

9. Snapshots

Each ingested leaderboard produces a per-day snapshot: model, org, score, CI bounds, and rank at the time of capture. Snapshots let a reader answer "who was on top on this board six months ago?" without needing to re-ingest historical data.

The snapshot job is idempotent and cadence-gated on MAX(snapshotDate) — no separate marker column. If today's snapshot for a board is already present, the job returns immediately; if not, it writes exactly one snapshot per (board, model, day). Deriving the cadence gate from the same table it writes to is cheaper to reason about than a parallel state column that can drift.

Snapshots are currently used to compute the coverage-shrinkage divisor and to power the arrival-timeline view. They are not surfaced as a stand-alone time-travel UI — that's a possible future addition, not a shipped feature. When it ships, it will land here.

10. Cross-axis insights

The /leaderboards/insights and per-model profile pages render editorial sentences. Every sentence is a computed fact from the current database; no sentence is authored by a language model in isolation. The pipeline has five stages.

1. Extract. Deterministic functions read the current database and produce InsightFinding structs — divergence, frontier, clustering, sensitivity, strength, weakness, standing, cohort, frontier-contribution. Every numeric field is rounded to display precision at extraction time.
2. Template. A deterministic renderer converts each struct into one plain English sentence. This template phrasing is always available and is what ships when the LLM stage fails.
3. LLM phrase. A low-temperature call to a small model reads the struct as JSON and returns a single sentence — a natural-language rewording. The prompt is strict: state only the facts in the struct, no qualifiers of magnitude, no superlatives, no causal or implicational language.
4. Validation gate. The candidate sentence must pass four checks before it ships (below).
5. Fallback. Any gate failure falls back to the template phrasing. The reader never sees a raw error, never sees a hallucination, and never sees a claim the struct didn't publish.

The four validation checks

• Number faithfulness. Every numeric token in the candidate sentence must map (within ±0.05 absolute, plus an integer-rounding fallback so "88.4" phrased as "88" is accepted) to a number that appears in the struct's declared allowed-numbers set. A number the struct did not publish is a hallucination and the sentence is rejected.
• Banned qualifiers. A curated list of 34 words and phrases that add magnitude, causation, or implication a struct doesn't carry ("dramatically", "cementing its lead", "which means") is checked case-insensitively. Any match rejects the sentence.
• Required mentions. Every model name and axis label the struct names must appear (case-insensitive, dash/space/underscore-normalized) in the sentence. A phrasing that omits its own subject is a different sentence, not a phrasing of the finding.
• Length cap. Phrasings longer than 320 characters are defensively rejected. A 400-character "insight" is either a run-on or a hallucinated aside.

Rounding contract. Percentages round to 1 decimal, correlations to 2, counts and ranks to integers. Rounding happens at extraction, before the LLM ever sees the struct — the gate compares against those rounded values, so LLM output that fabricates decimal precision the struct didn't publish gets rejected on the number-faithfulness check.

11. Coverage honesty

Thin-coverage models (appearing on fewer than 3 of the axes considered) trigger two visible consequences, not one.

• The standing finding sets a thin-coverage flag; both its template and its LLM phrasing include an explicit caveat that names the coverage ratio and the shrinkage. The reader sees the confidence-shape before reading any specific claim.
• Weakness findings are suppressed entirely for thin-coverage models. A bottom-percentile on 2 axes is a coverage artifact, not a measured weakness, and we refuse to render it as one. Strengths are still emitted — a high percentile on 2 axes is at least a positive claim about the axes actually measured — but they're read in the context of the thin-coverage caveat above.

Per-model profile thresholds: a strength requires a percentile of at least 85; a weakness requires a percentile of at most 15; a cohort finding requires at least 3 peers released within ±180 days of the model itself; a profile emits at most 6 findings, in a fixed order (standing → strengths → weaknesses → divergence → frontier contributions → cohort → sensitivity).

12. Glossary

Axis

One category chip — an LMArena subset, an Epoch benchmark, HHEM, MMLU-Pro, or Terminal-Bench 2. Every axis has one unit (Elo or percent) and one native population.

Percentile

0–100 rank within an axis's own population. Unitless. The composite blends percentiles, never raw scores.

Composite

Weight-normalized mean of per-axis percentiles across the axes a model appears on. Rendered as both the raw value and the coverage-shrunk value; rows sort on the shrunk one.

Shrinkage

The standard small-sample move — pull an estimate toward a neutral prior when there's little evidence. Here the prior is worth 3 axes at the neutral 50th percentile.

Dormant axis

An axis whose upstream source hasn't published new results in 6+ months. Still included in the blend if weighted, but visibly badged on every weight control so a stale-heavy blend can't hide as "current."

Divergence

Max minus min percentile across the axes a model actually appears on. Zero when the model is only on one axis.

Rank swing

The rank spread across the five built-in templates. Requires at least two templates to be defined; single-template rows are flagged rather than assigned zero.

Frontier

Running maximum of scores on one benchmark, in release-date order. Rises when a new model publishes a higher score than any prior model.

Cluster

A connected component on the correlation graph of axis percentile-vectors, edges at Pearson r ≥ 0.85. Not a semantic identity — two axes can correlate because model scale drives both.

Canonical key

The normalized identifier we use to match one physical model across multiple boards. Different boards write "Claude 3.5 Sonnet", "claude-3-5-sonnet", "claude-3.5-sonnet-20241022" — the canonical key collapses those to the same row on the leaderboard.

13. Known limitations

• Two candidate boards are not read from their own tables. MMMU's data is CC-BY-SA-4.0, a share-alike clause we cannot carry beside the other axes' more permissive licences, and the LiveCodeBench project's leaderboard publishes no licence we could find. Both stay as link-out cards on the Reference page. The LiveCodeBench column on this site is Artificial Analysis's own run of that benchmark, read from the CC-BY-4.0 Every Eval Ever datastore and labelled "LiveCodeBench (Artificial Analysis)" wherever it appears — the Reference card for LiveCodeBench says so and links to it.
• Dormant axes still contribute to blends when the reader (or a template) weights them. They're never silently filtered — the honesty move is to badge them everywhere, not to hide them.
• Rank swing is measured across the five built-in templates, not across all possible weight vectors. A user-supplied custom weighting isn't included in the swing figure — the figure is a discrete robustness check, not a continuous stability guarantee.
• On the current data, most adequately-covered axis pairs correlate positively. The clustering page reflects that honestly (one large cluster, a small number of singletons); it doesn't manufacture a tidy skill taxonomy the data doesn't support.
• Arrival timelines are limited to Epoch benchmarks because Epoch scores are fixed capability metrics with reliable release-date semantics. Adding LMArena or another Elo-based board would misrepresent the underlying data.
• No per-model score history. A model's benchmark score is fixed; charting it over time would imply the model itself was changing, which is false.
• Thin-coverage profiles (< 3 axes) show strengths but not weaknesses, and their standing carries an explicit caveat. Do not read a thin-coverage profile as a full assessment — that's exactly the shape it deliberately isn't.

14. Changelog

Changes to methodology land here, not silently in code. "Cosmetic" edits to the page copy don't warrant an entry; a change to any load-bearing constant, threshold, or ordering rule does.

• 2026-09-12 — Page created. All methodology-relevant constants are now referenced by import, not by hardcoded prose.

15. How to verify

Nothing on this site is a black box. Anyone can reproduce any composite or insight from the raw ingested data.

1. Start at the Sources page for a link to every upstream dataset. Every value we render is traceable back to a specific row in one of those datasets, subject to the modifications listed on the source card (subset selection, schema normalization).
2. To hand-check a composite: rank the model on each selected axis to get its percentiles, weight-average them, then apply the shrinkage (n · mean + 3 · 50) / (n + 3) where n is the model's coverage. The shrunk value is what the row sorts on; hovering the composite cell shows both.
3. To hand-check a divergence value: the model's highest and lowest percentile across the axes it actually appears on. Divergence is their difference; the divergence tooltip on the row names both axes explicitly.
4. To hand-check an insight sentence: click its "Verify" link. That link jumps to the specific /leaderboards view where the underlying numbers are visible. The sentence carries no fact the linked view doesn't show.

Insight prose that came from the deterministic template rather than the LLM is badged "template" on the card. That's an observability marker, not a quality signal — the template phrasing is the safe fallback and is always as truthful as the struct.