analiza aiqbench
This is a theoretical machine learning paper on recalibration of probability predictors across many related binary questions with few labels each. Its primary subject is statistical learning theory (minimax rates, identification limits) applied to calibration, so it passes the topic gate. It falls under Machine Learning, since the contribution is about estimators and sample-complexity bounds rather than a product, safety framework, or hardware. aiqbench take: The paper gives minimax rates and identification limits for recalibrating a shared predictor across many questions with few labels each, showing that pooling information with a per-question correction can match the optimal rate without knowing problem parameters, while per-question tuning can lose a logarithmic-type factor. Its main caveat is that the results are asymptotic and bound-based, and the practical relevance depends on whether real decision models satisfy its assumptions (such as the fixed-effects shape or the non-atomic output law). For readers building LLM-based probability tools, the actionable point is that pooling calibration data across questions is theoretically justified, but some questions may be fundamentally unidentifiable from their own labels.
Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits
arXiv:2610.18488v1 Announce Type: new Abstract: Many related prediction problems, which we call questions, come with few labels each. What information can be shared across them, and what is actually necessary for good decisions? We answer both for recalibration, motivated by decision models such as Jev (TypeSafe AI, 2026) that return a probability for any yes/no question a developer writes. A predictor reports a probability $P$ with output law $\mu$ on $m$ questions, each with $n$ labels and a nondecreasing map $g_j(p)=\mathbb{P}(Y=1\mid P=p)$; the compound risk is the excess Brier score over the true maps, averaged over questions. Over maps within sup-distance $\tau\le 1$ and total variation $V\le 2$ of one nondecreasing shape (fixed effects), the minimax compound risk is $(mn)^{-2/3}+\min\{\tau^2,1/n\}+\min\{\tau^2,V^2,(V/n)^{2/3}\}$ up to universal constants, for every non-atomic $\mu$. A pooled isotonic fit with a per-question correction sized once, on held-out labels of all questions, attains it without knowing $\tau$, $V$ or $\mu$ ($n\ge 2$, $m\ge 3$); tuning each question on its own labels loses a factor of order $\min\{n,\log(m^2/n)\}$ when the maps differ by constants and $m^2/n$ is large. When the maps are drawn from an unknown prior (random effects), other questions need not identify the Bayes rule. The loss, the identification regret, is at most $2^{-n}$ once the law of a question's data is known, extending a bound of Skibinsky (1968) for coins, and in the worst case at least $\beta_\star^{-n}/(72\sqrt{n})$ at every finite $m$, where $\beta_\star\approx 3.583$. Beyond a few labels per question, what cannot be learned is nearly unnecessary. For constant maps (an uninformative predictor) and constant-map priors, the exponential base is exactly $\beta_\star$, not the 4 suggested by the moment range; for monotone maps it lies in $[2,\beta_\star]$.
Abstract:Many related prediction problems, which we call questions, come with few labels each. What information can be shared across them, and what is actually necessary for good decisions? We answer both for recalibration, motivated by decision models such as Jev (TypeSafe AI, 2026) that return a probability for any yes/no question a developer writes. A predictor reports a probability $P$ with output law $\mu$ on $m$ questions, each with $n$ labels and a nondecreasing map $g_j(p)=\mathbb{P}(Y=1\mid P=p)$; the compound risk is the excess Brier score over the true maps, averaged over questions. Over maps within sup-distance $\tau\le 1$ and total variation $V\le 2$ of one nondecreasing shape (fixed effects), the minimax compound risk is $(mn)^{-2/3}+\min\{\tau^2,1/n\}+\min\{\tau^2,V^2,(V/n)^{2/3}\}$ up to universal constants, for every non-atomic $\mu$. A pooled isotonic fit with a per-question correction sized once, on held-out labels of all questions, attains it without knowing $\tau$, $V$ or $\mu$ ($n\ge 2$, $m\ge 3$); tuning each question on its own labels loses a factor of order $\min\{n,\log(m^2/n)\}$ when the maps differ by constants and $m^2/n$ is large. When the maps are drawn from an unknown prior (random effects), other questions need not identify the Bayes rule. The loss, the identification regret, is at most $2^{-n}$ once the law of a question's data is known, extending a bound of Skibinsky (1968) for coins, and in the worst case at least $\beta_\star^{-n}/(72\sqrt{n})$ at every finite $m$, where $\beta_\star\approx 3.583$. Beyond a few labels per question, what cannot be learned is nearly unnecessary. For constant maps (an uninformative predictor) and constant-map priors, the exponential base is exactly $\beta_\star$, not the 4 suggested by the moment range; for monotone maps it lies in $[2,\beta_\star]$.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2610.18488 [cs.LG]
(or arXiv:2610.18488v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.18488 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 374 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 3 oct. 2026
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · 8 oct. 2026
Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore
AWS · 16 oct. 2026