aiqbench analysis
This research addresses the critical safety gap in multi-agent LLM systems where consensus among agents can erroneously reinforce incorrect answers. By introducing Conformal Social Choice, the authors provide a post-hoc calibration layer that guarantees fixed-round coverage of the correct answer and explicitly flags cases requiring human escalation rather than autonomous action. The method demonstrates that while debate does not inherently improve reasoning accuracy, this statistical framework effectively makes system failures actionable and quantifiable.
From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
arXiv:2604.07667v2 Announce Type: replace Abstract: Multi-agent debate improves LLM reasoning, yet agreement is not evidence of correctness. When agents converge on a wrong answer through social reinforcement, consensus stopping commits that error to automated action with no recourse. We introduce Conformal Social Choice, a post-hoc layer that converts debate outputs into calibrated act-versus-escalate decisions. Verbalized probability distributions from heterogeneous agents are aggregated via a linear opinion pool and calibrated with split conformal prediction, yielding fixed-round marginal coverage: at any pre-specified round, the correct answer is included with probability ${\geq}\,1{-}\alpha$, without assumptions on individual model calibration. A hierarchical action policy maps singletons to autonomous action and larger sets to human escalation. Across eight MMLU-Pro domains and two additional benchmarks, fixed-round coverage stays close to target. The key finding is not that debate becomes more accurate, but that the conformal layer makes its failures actionable: at the final round, 81.9\% of wrong-consensus cases are flagged by non-singleton sets at $\alpha{=}0.05$. Separately, among cases that reach a singleton before the final round, accuracy reaches 90.0--96.8\% on MMLU-Pro (up to 22.1pp above consensus stopping): a selection effect, not a reasoning improvement. Simple probability- and entropy-threshold rules attain comparable interception; conformal's distinct value is a finite-sample fixed-round coverage guarantee together with explicit candidate sets for escalation. Adaptive first-singleton stopping is evaluated empirically and not covered by this guarantee.
Abstract:Multi-agent debate improves LLM reasoning, yet agreement is not evidence of correctness. When agents converge on a wrong answer through social reinforcement, consensus stopping commits that error to automated action with no recourse. We introduce Conformal Social Choice, a post-hoc layer that converts debate outputs into calibrated act-versus-escalate decisions. Verbalized probability distributions from heterogeneous agents are aggregated via a linear opinion pool and calibrated with split conformal prediction, yielding fixed-round marginal coverage: at any pre-specified round, the correct answer is included with probability ${\geq}\,1{-}\alpha$, without assumptions on individual model calibration. A hierarchical action policy maps singletons to autonomous action and larger sets to human escalation. Across eight MMLU-Pro domains and two additional benchmarks, fixed-round coverage stays close to target. The key finding is not that debate becomes more accurate, but that the conformal layer makes its failures actionable: at the final round, 81.9\% of wrong-consensus cases are flagged by non-singleton sets at $\alpha{=}0.05$. Separately, among cases that reach a singleton before the final round, accuracy reaches 90.0--96.8\% on MMLU-Pro (up to 22.1pp above consensus stopping): a selection effect, not a reasoning improvement. Simple probability- and entropy-threshold rules attain comparable interception; conformal's distinct value is a finite-sample fixed-round coverage guarantee together with explicit candidate sets for escalation. Adaptive first-singleton stopping is evaluated empirically and not covered by this guarantee.
Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Social and Information Networks (cs.SI)
arXiv:2604.07667 [cs.AI]
(or arXiv:2604.07667v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2604.07667 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 318 words
Related coverage
More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration
arXiv · Jun 8, 2026
Agent Collectives Should Not Detect Their Own Imposters: A Chess Case Study
arXiv · Sep 30, 2026
AGORA: Can Deliberation and Governance Gates Absorb Participation Bias in Transit Planning?
arXiv · Jun 15, 2026
Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layer
arXiv · Sep 18, 2026