analiza aiqbench
This arXiv paper studies whether LLMs communicate pragmatically in multiparty collaborative tasks where participants hold different private information, formalizing the problem through Grice's cooperative principle. Its main finding is measured rather than alarmist: models show some pragmatic competence but still struggle when information is incomplete, with failures that map onto specific Gricean maxims. For readers evaluating LLMs as "collaborators," the practical takeaway is that agent benchmarks should test communication under asymmetric information, not just task completion with full context.
Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry
arXiv:2607.11053v2 Announce Type: replace-cross Abstract: Fruitful collaborations require effective communication of privately-held information. The increasing use of LLMs as "collaborators" raises questions about the extent to which they exhibit these pragmatic capabilities. In this paper, we perform a novel investigation into the pragmatic reasoning capabilities of LLMs in a multiparty collaborative task under partial information conditions. We first formalize a notion of collaborative epistemic asymmetry and prove that fully actionable instructions in this context requires adherence to Grice's cooperative principle. We then empirically assess various LLMs' abilities to act cooperatively as both speakers and listeners, including both prompting and post-training strategies. Our results show that while LLMs exhibit certain pragmatic capabilities in collaborative settings, they still face challenges in pragmatic communication with incomplete information, and that Grice's maxims predict distinct failure modes.
Abstract:Fruitful collaborations require effective communication of privately-held information. The increasing use of LLMs as "collaborators" raises questions about the extent to which they exhibit these pragmatic capabilities. In this paper, we perform a novel investigation into the pragmatic reasoning capabilities of LLMs in a multiparty collaborative task under partial information conditions. We first formalize a notion of collaborative epistemic asymmetry and prove that fully actionable instructions in this context requires adherence to Grice's cooperative principle. We then empirically assess various LLMs' abilities to act cooperatively as both speakers and listeners, including both prompting and post-training strategies. Our results show that while LLMs exhibit certain pragmatic capabilities in collaborative settings, they still face challenges in pragmatic communication with incomplete information, and that Grice's maxims predict distinct failure modes.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2607.11053 [cs.CL]
(or arXiv:2607.11053v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.11053 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 223 cuvinte
Subiecte conexe
Fourth Forum on China - ASEAN Artificial Intelligence Cooperation Successfully Held in Nanning, Drawing Over 700 Delegates
vietnamnews.vn · 20 sept. 2026
When Contextual Inference Fails: Cancelability in Interactive Instruction Following
arXiv · 21 aug. 2026
XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?
arXiv · 9 sept. 2026
Xi Jinping vision for artificial intelligence - How he will bridge the gap with US tech giants
bankingnews.gr · 25 aug. 2026