aiqbench analysis
This large-scale study tests 284 interpretable linguistic features across 30 LLMs and 10 text domains, and finds that classifiers built on these features can separate AI-generated from human text. Most individual indicators, however, depend heavily on context, with lexical richness the notable exception that holds across model families and domains. For researchers and practitioners, the useful takeaway is that only a small set of signals generalizes, so explanations of "machine-generated" text should not rely on features that work in one setting alone.
A Systematic Analysis of Linguistic Features in AI-Generated English Text Detection Across Domains and Models
arXiv:2606.04177v2 Announce Type: replace-cross Abstract: Interpretable linguistic features offer a promising approach for explaining why a given text appears machine-generated, particularly for non-expert users. However, existing findings on which features reliably indicate LLMgenerated text remain fragmented across feature sets, models, and text domains. To address this gap, we conduct a large-scale empirical study assessing the robustness of linguistic signals for characterizing AI-generated text. Our analysis covers 284 interpretable linguistic features across outputs from 30 LLMs and 10 text domains under cross-model and cross-domain generalization settings. We show that classifiers based solely on linguistic features can reliably distinguish AI-generated from human-written text. However, many previously proposed indicators prove strongly context-dependent, with the exception of measures of lexical richness, which remain robust signals across model families and text domains. These results demonstrate which linguistic signals generalize across contexts and provide a foundation for more reliable, interpretable analyses of AI-generated language.
Abstract:Interpretable linguistic features offer a promising approach for explaining why a given text appears machine-generated, particularly for non-expert users. However, existing findings on which features reliably indicate LLMgenerated text remain fragmented across feature sets, models, and text domains. To address this gap, we conduct a large-scale empirical study assessing the robustness of linguistic signals for characterizing AI-generated text. Our analysis covers 284 interpretable linguistic features across outputs from 30 LLMs and 10 text domains under cross-model and cross-domain generalization settings. We show that classifiers based solely on linguistic features can reliably distinguish AI-generated from human-written text. However, many previously proposed indicators prove strongly context-dependent, with the exception of measures of lexical richness, which remain robust signals across model families and text domains. These results demonstrate which linguistic signals generalize across contexts and provide a foundation for more reliable, interpretable analyses of AI-generated language.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.04177 [cs.CL]
(or arXiv:2606.04177v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.04177 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words