analiza aiqbench
This research addresses the critical challenge of auditing Large Language Model calibration when commercial APIs obscure internal probability outputs. By exploiting the logit_bias parameter, the authors develop a method to estimate true calibration error using only a single query per sample, effectively bypassing API opacity. This technique offers a practical framework for verifying the reliability of black-box foundation models without requiring access to raw logits.
Single-Query Black-Box Calibration Auditing via Logit Bias
arXiv:2609.05125v1 Announce Type: new Abstract: Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.
Abstract:Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.
Machine Learning (cs.LG)
arXiv:2609.05125 [cs.LG]
(or arXiv:2609.05125v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.05125 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 186 cuvinte
Subiecte conexe
What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
EleutherAI · 25 aug. 2026
A Key-Position Confound in no-cot-bench (and implications for interpreting looped transformers)
LessWrong · 2 oct. 2026
[Paper] Distillation for Incrimination and Distillation for Capabilities
LessWrong · 9 oct. 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · 21 iul. 2026