Communication styles and reader preferences of LLM- and human-authored COVID-19 information explanations: a case study

Important: e-prints posted on arXiv are not peer-reviewed by arXiv; they should not be relied upon without context to guide clinical practice or health-related behavior and should not be reported in news media as established information without consulting multiple experts in the field.

Abstract:With the wide adoption of large language models (LLMs) in information assistance, it is essential to examine their alignment with human communication styles and values. We situate this study within health fact-checking, where effective communication is critical for correcting misconceptions and building trust. Although recent studies have explored LLMs for fact-checking and health communication, differences between LLM and human communication styles and associated reader perceptions remain under-explored. We compiled a dataset of 1,498 health misinformation claims and explanations from authoritative fact-checking organizations and generated LLM responses to inaccurate health information. Drawing on health communication theories, we evaluated communication styles across three dimensions: information linguistic features, sender persuasive strategies, and receiver value alignments. We further assessed reader perceptions through a blinded evaluation with 99 participants. LLM-generated articles scored significantly lower in persuasive strategies, certainty expressions, and alignment with social values and moral foundations. However, participants strongly preferred LLM content, with over 60% of responses favoring LLM articles for clarity, completeness, and persuasiveness. This preference was associated with structured presentation, clarity, and neutral tone, which may convey completeness and professionalism despite reduced nuance or rigor. These findings highlight both the potential and limitations of LLMs in health communication and fact-checking, suggesting that reader preference may not necessarily correspond to established measures of communication quality.

Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI)

arXiv:2505.08143 [cs.HC]

(or arXiv:2505.08143v2 [cs.HC] for this version)

https://doi.org/10.48550/arXiv.2505.08143 Focus to learn more arXiv-issued DOI via DataCite

https://doi.org/10.1186/s44398-026-00030-z Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.