aiqbench analysis
CORDIAL introduces a novel calibration method that treats LLM probability distributions as noisy measurements, correcting them with a minimal set of interpretable parameters derived from very few labels. This approach significantly reduces log loss across diverse benchmarks while preserving stochastic order, offering an efficient alternative to data-hungry baselines for improving model reliability in ordinal tasks.
CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels
arXiv:2609.29807v1 Announce Type: cross Abstract: A large language model (LLM) can turn a text into a distribution over an ordered scale, but that distribution is a noisy measurement: saturated, compressed or exaggerated, and biased in a consistent direction. We propose CORDIAL, which treats the model's output as a noisy reading of the true label and corrects it with a channel of five interpretable parameters. The channel is small enough for its posterior to be averaged from a handful of labels, and we prove that the resulting calibration preserves first-order stochastic order. On Amazon reviews and CMU-MOSEI transcripts with four LLMs, CORDIAL has the lowest log loss among nine calibrators in 76 of 80 settings with 5 to 100 labels; with 20 labels and the main 7B reader, it matches the strongest baseline using 28-54 labels. The same posterior lets us learn priors from other tasks and fuse several LLMs. Unrestricted calibrators such as Dirichlet calibration overtake it only as the calibration set grows into the hundreds or thousands.
Abstract:A large language model (LLM) can turn a text into a distribution over an ordered scale, but that distribution is a noisy measurement: saturated, compressed or exaggerated, and biased in a consistent direction. We propose CORDIAL, which treats the model's output as a noisy reading of the true label and corrects it with a channel of five interpretable parameters. The channel is small enough for its posterior to be averaged from a handful of labels, and we prove that the resulting calibration preserves first-order stochastic order. On Amazon reviews and CMU-MOSEI transcripts with four LLMs, CORDIAL has the lowest log loss among nine calibrators in 76 of 80 settings with 5 to 100 labels; with 20 labels and the main 7B reader, it matches the strongest baseline using 28-54 labels. The same posterior lets us learn priors from other tasks and fuse several LLMs. Unrestricted calibrators such as Dirichlet calibration overtake it only as the calibration set grows into the hundreds or thousands.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2609.29807 [cs.CL]
(or arXiv:2609.29807v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.29807 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection
arXiv · Sep 22, 2026
TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
arXiv · Oct 10, 2026
Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth
arXiv · Oct 8, 2026
A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs
arXiv · Sep 7, 2026