Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

arXiv:2609.37470v1 Announce Type: new Abstract: A probability used for a decision should refer to the same event across equivalent requests. We introduce probability contracts, a benchmark connecting exact finite-world posteriors, validated event transformations, and failure-aware decision evaluation. Four model-interface configurations are evaluated on 1,000 worlds. Their assessments differ across accuracy, coherence, and decision loss: Kev has lower aggregate canonical posterior error than Jev, but larger complement and coarsening residuals, with accuracy ordering varying by stratum. Jev's Event and Choice interfaces induce different binary actions on 32.8% of valid pairs at defer cost 0.10. A post-hoc analysis finds that disagreement certifies only 11-52% of mean binary pair error across configurations. An elementary action-region characterization explains when averaging changes decision loss relative to randomly selecting one interface. Although averaging cannot worsen that baseline's expected Brier score, its decision effect depends on the cost and crossed boundaries; the observed same-baseline penalties occur in configurations already worse than always deferring. The benchmark makes these distinctions measurable without treating consistency as accuracy or a score improvement as a decision guarantee.

Abstract:A probability used for a decision should refer to the same event across equivalent requests. We introduce probability contracts, a benchmark connecting exact finite-world posteriors, validated event transformations, and failure-aware decision evaluation. Four model-interface configurations are evaluated on 1,000 worlds. Their assessments differ across accuracy, coherence, and decision loss: Kev has lower aggregate canonical posterior error than Jev, but larger complement and coarsening residuals, with accuracy ordering varying by stratum. Jev's Event and Choice interfaces induce different binary actions on 32.8% of valid pairs at defer cost 0.10. A post-hoc analysis finds that disagreement certifies only 11-52% of mean binary pair error across configurations. An elementary action-region characterization explains when averaging changes decision loss relative to randomly selecting one interface. Although averaging cannot worsen that baseline's expected Brier score, its decision effect depends on the cost and crossed boundaries; the observed same-baseline penalties occur in configurations already worse than always deferring. The benchmark makes these distinctions measurable without treating consistency as accuracy or a score improvement as a decision guarantee.

Machine Learning (cs.LG); Machine Learning (stat.ML)

arXiv:2609.37470 [cs.LG]

(or arXiv:2609.37470v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.37470 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

Subiecte conexe