The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

Abstract:Why does machine-generated text remain detectable? We trace the answer to the decoding stage: standard strategies such as top-$k$ and nucleus sampling restrict generation to high-probability tokens, while human writers routinely choose words that are contextually apt yet rank low under a model's next-token distribution. Truncation therefore renders a measurable share of human word choices unreachable; we call this set the truncation blind spot. Across five open models and three domains, 8--18\% of human-selected tokens fall outside common truncation boundaries, with content words excluded at 2.9 times the rate of function words. As a consequence, simple classifiers using two features separate 1.8 million machine generations from human text at mean AUC-ROC near 0.97, and detectability tracks truncation intensity rather than model scale, architecture, or alignment. Probability-floor samplers substantially narrow the blind spot, and generation experiments confirm the exclusion--detectability link out of sample. Code, data, and the measurement harness are released at this https URL.

Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML)

arXiv:2603.18482 [cs.CL]

(or arXiv:2603.18482v3 [cs.CL] for this version)