aiqbench analysis
This arXiv preprint extends a matrix-algebra framework for classifier evaluation from discrimination metrics (precision, recall, F1, MCC) to calibration (Brier score, log-loss) and ranking (nDCG, AUC). Its most concrete claims concern which metrics are skew-invariant or strictly proper, and some results are machine-checked in Lean 4. Practitioners choosing evaluation metrics for multilabel or ranking systems should find the properness and aggregation distinctions useful, though the experiments are limited to two benchmarks.
Discrimination, Calibration, and Ranking: A Unified Matrix Framework
arXiv:2611.02215v1 Announce Type: cross Abstract: Discrimination, calibration, and ranking are the three pillars of classifier evaluation, yet no single algebraic language spans all three. A recent indicator-matrix account unifies discrimination measures (precision, recall, F1, Matthews correlation) across binary, multiclass, multilabel and related settings, using Boolean cell operations on indicator label matrices, generalised to soft labels by t-norms. Calibration measures (Brier score, log-loss) and ranking measures (nDCG, AUC) fall outside it. We close the gap with two further classes of cell operators: deviation operators phi(y - yhat), which give a Calibration Framework, and rank- and slice-dependent operators (positional gains for DCG, column concordance cells for AUC), which give a Ranking Framework. Both inherit the micro/macro/exemplar aggregation architecture. For calibration, the three Brier aggregations collapse to one scalar; the Murphy decomposition holds in matrix form (unbinned, binned and isotonic); and, within the Brier-type class-indexed family, a measure is skew-invariant if and only if it depends only on the class-conditional errors alpha_j and beta_j, the calibration analogues of recall and specificity. For ranking, raw DCG collapses across aggregations while nDCG does not; per-class nDCG and per-class AUC are related by a row-column duality; micro-AUC differs from pooled ROC-AUC; raw positional scores are never strictly proper, and nDCG and AUC need not be proper. The skew-invariant per-class building blocks share one class-conditional expectation template. Experiments on two multilabel benchmarks illustrate the distinctions. Many of the results are machine-checked in Lean 4; the proofs, the experiments and the aggmetrics.scores package are public: https://github.com/rcprati/discrimination-calibration-ranking
Abstract:Discrimination, calibration, and ranking are the three pillars of classifier evaluation, yet no single algebraic language spans all three. A recent indicator-matrix account unifies discrimination measures (precision, recall, F1, Matthews correlation) across binary, multiclass, multilabel and related settings, using Boolean cell operations on indicator label matrices, generalised to soft labels by t-norms. Calibration measures (Brier score, log-loss) and ranking measures (nDCG, AUC) fall outside it. We close the gap with two further classes of cell operators: deviation operators phi(y - yhat), which give a Calibration Framework, and rank- and slice-dependent operators (positional gains for DCG, column concordance cells for AUC), which give a Ranking Framework. Both inherit the micro/macro/exemplar aggregation architecture. For calibration, the three Brier aggregations collapse to one scalar; the Murphy decomposition holds in matrix form (unbinned, binned and isotonic); and, within the Brier-type class-indexed family, a measure is skew-invariant if and only if it depends only on the class-conditional errors alpha_j and beta_j, the calibration analogues of recall and specificity. For ranking, raw DCG collapses across aggregations while nDCG does not; per-class nDCG and per-class AUC are related by a row-column duality; micro-AUC differs from pooled ROC-AUC; raw positional scores are never strictly proper, and nDCG and AUC need not be proper. The skew-invariant per-class building blocks share one class-conditional expectation template. Experiments on two multilabel benchmarks illustrate the distinctions. Many of the results are machine-checked in Lean 4; the proofs, the experiments and the this http URL package are public: this https URL
88 pages, 8 figures. Lean 4 formalisation and experiment code: this https URL. Implemented in the aggmetrics package (version 0.3.0, subpackage this http URL)
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2611.02215 [cs.LG]
(or arXiv:2611.02215v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2611.02215 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 292 words
Related coverage
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
arXiv · Oct 15, 2026
Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods
arXiv · Oct 19, 2026
The C-index illusion: discrimination without calibration in published survival models
arXiv · Jul 23, 2026
A Longitudinal Attribute-Conditioned Neural Network for Modeling Health-State Transition Probabilities in Temporally Irregular Data: The LANTERN Framework
arXiv · Jun 15, 2026