aiqbench analysis
CalBench introduces a critical benchmark for evaluating multi-agent LLMs in scenarios requiring strict privacy preservation, revealing that simple task completion often masks significant failures in cost efficiency and burden fairness. The study demonstrates that communication volume does not correlate with reduced regret and that excessive privacy silence can hinder collaborative optimization, challenging the assumption that non-disclosure is always optimal. This work provides an essential testbed for assessing whether autonomous assistants can safely coordinate on behalf of users before large-scale deployment.
CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs
arXiv:2605.09823v3 Announce Type: replace-cross Abstract: Personal AI assistants are beginning to act as delegates with access to calendars, inboxes, and user preferences. Calendar scheduling makes the trust problem concrete: an assistant must coordinate with other assistants while deciding what to reveal about the person it represents. We introduce CalBench, a controlled benchmark for multi-agent calendar scheduling under private information. In each task, $N$ agents manage separate private calendars and schedule a stream of $M$ incoming meetings while minimizing disruption costs. Because no agent can inspect another agent's calendar, success requires language-mediated coordination rather than centralized planning. CalBench generates solvable scenarios with CP-SAT oracle solutions and decentralized non-LLM reference protocols, enabling evaluation of task success, excess cost, communication efficiency, burden fairness, and privacy leakage under matched information constraints. Across seven model families, we find that completion alone misses important failures: agents leave avoidable cost on the table, communication volume does not predict lower regret, and privacy-preserving silence can deprive teammates of cost information needed for fair burden allocation. CalBench provides a reproducible testbed for studying whether autonomous assistants can coordinate on behalf of users before deployment at scale.
Abstract:Personal AI assistants are beginning to act as delegates with access to calendars, inboxes, and user preferences. Calendar scheduling makes the trust problem concrete: an assistant must coordinate with other assistants while deciding what to reveal about the person it represents. We introduce CalBench, a controlled benchmark for multi-agent calendar scheduling under private information. In each task, $N$ agents manage separate private calendars and schedule a stream of $M$ incoming meetings while minimizing disruption costs. Because no agent can inspect another agent's calendar, success requires language-mediated coordination rather than centralized planning. CalBench generates solvable scenarios with CP-SAT oracle solutions and decentralized non-LLM reference protocols, enabling evaluation of task success, excess cost, communication efficiency, burden fairness, and privacy leakage under matched information constraints. Across seven model families, we find that completion alone misses important failures: agents leave avoidable cost on the table, communication volume does not predict lower regret, and privacy-preserving silence can deprive teammates of cost information needed for fair burden allocation. CalBench provides a reproducible testbed for studying whether autonomous assistants can coordinate on behalf of users before deployment at scale.
Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI)
arXiv:2605.09823 [cs.MA]
(or arXiv:2605.09823v3 [cs.MA] for this version)
Full text under the publisher's open licence · 196 words
Related coverage
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning
arXiv · Sep 24, 2026
Agentic Trading: When LLM Agents Meet Financial Markets
arXiv · Oct 7, 2026