DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving

Abstract:Multi-party dialogue is a critical setting for studying collaborative reasoning and decision-making, yet existing datasets rarely focus on structured, reasoning-intensive tasks. We introduce DeliChess, a dataset of group deliberation dialogues in which participants collaboratively solve multiple-choice chess puzzles. Participants first answer independently, then engage in multi-party deliberation and revise their individual answers. The dataset comprises 107 dialogues with full transcripts, pre- and post-deliberation choices, and utterance-level annotations of communicative function, epistemic stance, and usefulness for supporting deliberation. Our analyses show that greater diversity in initial solution quality is associated with larger gains, while answer trajectories reveal how deliberation can recover, discover, or lose strong answers. Cases in which groups surpass every independent answer are associated with sustained reasoning and epistemic openness. We further propose a diagnostic action-selection task and find that the tested LLMs show only weak agreement with human-attested helpful actions. Together, our dataset provides a testbed for modelling group reasoning, dialogue dynamics, and conditions for effective deliberation.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)

arXiv:2606.04987 [cs.CL]

(or arXiv:2606.04987v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.04987 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.