MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

Abstract:Foundation-model pools are increasingly used as black-box responders in coordinated systems where a coordinator must decide which response to trust. Raw self-reported confidence is the natural signal, but is not comparable across models and becomes stale under distribution shift when corrected only at design time. We study runtime confidence calibration for multi-model coordination, where per-model corrections are learned online from deployment outcomes with no model access, no held-out calibration data, and no retraining. Across 18 open-weight foundation models, 8 benchmarks, and over 44,000 observations, we find that online adaptation is a family property: simple same-information online calibrators close most of the calibration gap left by frozen design-time methods under shift, and the forgetting schedule is the dominant design axis. We present MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a structured member of this family that maintains per-model, per-confidence-band multiplicative factors using symmetric exponentially weighted updates and shrinkage blending. MARGIN does not dominate the online family on expected calibration error (ECE) under abrupt shift. Its value lies in interpretable confidence-band trust factors, defined cold-start and returning-model behaviour, dynamic-pool support, and a scoped symmetric-update guarantee for fixed-policy non-strategic agents. Empirically, raw verbalized confidence is a weak or misleading pairwise selection signal on hard code-generation tasks, while online calibration substantially improves pairwise resolution and multi-model selection. We also evaluate delayed and selected-answer-only feedback; the latter materially degrades every same-information online method, MARGIN included. Runtime calibration acts as a coordination layer for heterogeneous foundation-model pools, and MARGIN is a practical inspectable instantiation.

Machine Learning (cs.LG); Multiagent Systems (cs.MA)

arXiv:2605.22949 [cs.LG]

(or arXiv:2605.22949v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.22949 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.