Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs

Abstract:We propose a methodology for investigating the differences between natural and synthetic language in a challenging setting: humorous improvisational dialogue. Our experimental setting compares human-produced comedy improvisations with the dialogues generated by LLMs in a multi-agent environment that we designed. The analysis combines automated linguistic measures with manual annotation grounded in Systemic Functional Linguistics (SFL). The results show that, although LLMs can create coherent and creative scenes, they often fail to reproduce key characteristics of spontaneous spoken interaction. In addition, LLM-generated interactions display a strong tendency toward politeness and positive evaluations, contrasting with the broader range of attitudes found in human humorous improvisation. Our evidence highlights both the strengths and limitations of current LLMs in modeling dialogic humor and demonstrate the value of integrating linguistic theory with NLP methods for the analysis of machine-generated language

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2611.00492 [cs.CL]

(or arXiv:2611.00492v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2611.00492 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

Springer Nature Link. MICAI 2026, CCIS,Part I proceeding

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.