analiza aiqbench
This arXiv paper evaluates three decision models on MetaTool, HotpotQA, and ECtHR, and finds that their output probabilities often misallocate mass across multiple correct choices. Even when the correct choices are ranked above all distractors, some receive less probability than a uniform 1/n split, and identical copies of one correct choice get very unequal scores. For practitioners who threshold model probabilities to select answers, the result is that a cutoff can rank correct options but cannot reliably recover the full correct set.
When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models
arXiv:2610.29217v1 Announce Type: cross Abstract: Decision models return normalized probability distributions over candidate choices, which downstream applications can use to select among choices. Although their providers emphasize the reliability of these probabilities, calibration on tasks with a single correct choice does not establish their reliability when multiple choices are correct. We evaluate three decision models (Jev, Liquid d1, and GPT-6 Luna Decisions) on three diverse datasets (MetaTool, HotpotQA, and ECtHR) to study probability allocation and thresholding. Even when models rank every correct choice above every distractor, in 28% to 50% of those lists, at least one correct choice receives a probability below the uniform probability of 1/n, where n is the number of choices. We also present each model with five identical copies of a correct choice and observe probability disparities, with mean gaps ranging from 17 to 79 percentage points. We further show how these allocation disparities affect the recovery of multiple correct choices under probability thresholding. Decision-model probabilities can rank correct choices, but a probability threshold does not identify the set of correct choices.
Abstract:Decision models return normalized probability distributions over candidate choices, which downstream applications can use to select among choices. Although their providers emphasize the reliability of these probabilities, calibration on tasks with a single correct choice does not establish their reliability when multiple choices are correct. We evaluate three decision models (Jev, Liquid d1, and GPT-6 Luna Decisions) on three diverse datasets (MetaTool, HotpotQA, and ECtHR) to study probability allocation and thresholding. Even when models rank every correct choice above every distractor, in 28% to 50% of those lists, at least one correct choice receives a probability below the uniform probability of 1/n, where n is the number of choices. We also present each model with five identical copies of a correct choice and observe probability disparities, with mean gaps ranging from 17 to 79 percentage points. We further show how these allocation disparities affect the recovery of multiple correct choices under probability thresholding. Decision-model probabilities can rank correct choices, but a probability threshold does not identify the set of correct choices.
11 pages, 2 figures. Code: this https URL
Artificial Intelligence (cs.AI)
arXiv:2610.29217 [cs.AI]
(or arXiv:2610.29217v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.29217 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 273 cuvinte
Subiecte conexe
University of Gda Team Identifies Permutation Asymmetry for Detecting Measurement Correlations
Quantum Zeitgeist · 16 iul. 2026
TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
arXiv · 10 oct. 2026
Exact Recovery by Neighborhood Smoothing in Directed Stochastic Block Models
arXiv · 15 sept. 2026
Selective Disclosure of Hidden Directives in Reasoning Models: Behavioral Asymmetry and Steering
arXiv · 1 sept. 2026