What Confidence Routing Is Actually Doing: Auditing Routing, Calibration, and Commitment in Multi-Agent Deliberation

Abstract:A common multi-agent design asks agents to report confidence and lets the highest-scoring agent speak next, implicitly using one scalar both to route the conversation and to estimate uncertainty. We audit this confidence-routed broadcast protocol by separating three trace-level questions: whether it selects the right candidate (routing), whether reported confidence behaves like a probability (calibration), and whether the selected agent publicly states the answer that won the turn (commitment). Our primary study covers 4,181 gpt-oss-120b olympiad-math traces; we repeat the audit on a 2-by-2 actor-by-benchmark grid that adds gemma-4-31B-it and a biology multiple-choice benchmark. In the primary cell, confidence discriminates correct from wrong candidates (AUROC 0.72) but is strongly overconfident (79% mean stated confidence versus 52% accuracy). A cross-fitted, tier-stratified isotonic procedure reduces Expected Calibration Error from 0.278 to 0.008 on held-out candidates, but it does not recover missing discrimination: raw AUROC is only 0.537 and 0.440 in the two Gemma cells. Routing is likewise setting-dependent. Fixed routers differ by at most 1.1 percentage points on gpt-oss/math, whereas raw-confidence argmax performs 5.6 and 11.2 points below random-valid selection in the Gemma cells. Commitment is distinct again: in the primary cell, poll and spoken answers diverge in 20.4% of valid pairs, 62.4% of those revisions are fresh generations, and the unconditional correctness shift is -1.7 points; the other three cells instead range from +0.9 to +12.2 points. The transferable lesson is procedural: routing discrimination, probability calibration, and public commitment must be measured separately before raw confidence is used for deployment decisions.

Multiagent Systems (cs.MA); Computation and Language (cs.CL)

arXiv:2609.27822 [cs.MA]

(or arXiv:2609.27822v1 [cs.MA] for this version)

https://doi.org/10.48550/arXiv.2609.27822 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.