aiqbench analysis
This study finds that participants could not reliably tell human-written news from LLM-generated news across six models, including 7B open-weight ones. Domain expertise improved accuracy, political orientation did not, and performance degraded after about 30 evaluations. For AI readers, the practical takeaway is that user-side detection is unlikely to be a dependable defense against synthetic misinformation, which points toward system-level measures such as content provenance. The sample is a single study on news text, so generalization to other formats and detection tools will need separate evidence.
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
arXiv:2604.03755v2 Announce Type: replace-cross Abstract: Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.
Title:Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
Abstract:Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.
6 pages, 6 figures, 1 table. Author version of the paper published in WebSci Companion '26, pp. 34-39
Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC)
H.5.2; H.1.2; K.4.2; I.2.7
arXiv:2604.03755 [cs.CY]
(or arXiv:2604.03755v2 [cs.CY] for this version)
https://doi.org/10.48550/arXiv.2604.03755 Focus to learn more arXiv-issued DOI via DataCite
Companion Publication of the 2026 18th ACM Web Science Conference (WebSci Companion '26), pp. 34-39
https://doi.org/10.1145/3795513.3807431 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 328 words