DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

Abstract:We introduce DiscoTrace, a method to identify the rhetorical strategies answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. The rich, community-sensitive answering behavior structurally revealed by DiscoTrace can guide the development of pragmatic LLM answerers that are more attuned to contextual information needs.

Computation and Language (cs.CL)

arXiv:2604.15140 [cs.CL]

(or arXiv:2604.15140v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2604.15140 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.