aiqbench analysis
Composing insight from the full article — this may take a moment.
Same Origin, Different Verdicts: Text Production Conditions Confound Machine-Generated Text Detection in Russian
arXiv:2611.05385v1 Announce Type: new Abstract: Automated detection of machine-generated text is applied where the verdict carries a cost for the author, yet origin is a property of the production process, not an observable property of the text. The conditions under which a text is produced push it in the same direction in which a generative model is presumed to push it. This study measures the contribution of these conditions while origin is held fixed. The Russian-language corpus of 1882 documents comprises a machine part, generated by four generation channels (each backed by its own model family) across 45 tasks in three task modes, and an archival human part annotated for source regimentation level before the texts were read. The primary contrast is computed within the machine corpus: both halves of a pair are generated by the same model for the same task and differ only in task wording. Machine-likeness is measured by four operationalisations: an interpretable index with weights fixed before data collection, a trained classifier, a zero-shot metric on an open model, a judge model. The main result is disagreement between procedures: changing the task mode shifted the scores of all four, but three gave a negative shift, the judge a positive one, and none of the four intervals covers zero. The comparison plan and the specifications of procedures 2-4 were fixed after the result of procedure 1, before procedures 2-4 ran; the disagreement survived correction of a corpus defect and of the nested validation. A baseline on the source identifier alone reaches a median AUROC of 1.0000 on the thirteen group splits where it is defined, and removing emphasis markers flipped a confident machine verdict to human without changing the full text by a single character. The scores are properties of a specific measurement system and corpus: they reflect data collection, repetition, formatting, and preprocessing decisions as well as linguistic style.
Abstract:Automated detection of machine-generated text is applied where the verdict carries a cost for the author, yet origin is a property of the production process, not an observable property of the text. The conditions under which a text is produced push it in the same direction in which a generative model is presumed to push it. This study measures the contribution of these conditions while origin is held fixed. The Russian-language corpus of 1882 documents comprises a machine part, generated by four generation channels (each backed by its own model family) across 45 tasks in three task modes, and an archival human part annotated for source regimentation level before the texts were read. The primary contrast is computed within the machine corpus: both halves of a pair are generated by the same model for the same task and differ only in task wording. Machine-likeness is measured by four operationalisations: an interpretable index with weights fixed before data collection, a trained classifier, a zero-shot metric on an open model, a judge model. The main result is disagreement between procedures: changing the task mode shifted the scores of all four, but three gave a negative shift, the judge a positive one, and none of the four intervals covers zero. The comparison plan and the specifications of procedures 2-4 were fixed after the result of procedure 1, before procedures 2-4 ran; the disagreement survived correction of a corpus defect and of the nested validation. A baseline on the source identifier alone reaches a median AUROC of 1.0000 on the thirteen group splits where it is defined, and removing emphasis markers flipped a confident machine verdict to human without changing the full text by a single character. The scores are properties of a specific measurement system and corpus: they reflect data collection, repetition, formatting, and preprocessing decisions as well as linguistic style.
65 pages, 6 figures, 24 tables. English translation of the frozen Russian manuscript; the Russian text is the source of record and is available with all data, code and run manifests at this https URL (v1.0.1)
Computation and Language (cs.CL)
arXiv:2611.05385 [cs.CL]
(or arXiv:2611.05385v1 [cs.CL] for this version)
Full text under the publisher's open licence · 356 words