aiqbench analysis
This arXiv paper is squarely about Large Language Models, so it passes the topic gate. It tests whether LLM string probabilities can separate grammatical from ungrammatical sentences, and finds that ungrammatical sentences do not show a distinct surprisal signature; semantically odd sentences in fact produce higher surprisal. The takeaway for readers is methodological: claims that models "understand" grammatical possibility based on probability alone need stronger evidence, since probability differences don't cleanly map onto syntactic knowledge.
Large Language Model probabilities cannot reliably distinguish between possible and impossible language
arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to the models sensitivity to what crosses the limits of grammatically impossible language, this evidence has been contested on the grounds of the soundness of the testing material. We use model-internal representations as a direct means of investigating the way Large Language Models represent the grammatical-ungrammatical distinction. In a novel benchmark, we elicit sum log probabilities from 8 models and compute minimal-pair surprisal differences, juxtaposing probabilities assigned to grammatical sentences to probabilities assigned to (i) lower frequency grammatical sentences, (ii) ungrammatical sentences, (iii) semantically odd sentences, and (iv) pragmatically odd sentences. The prediction is that if string-probabilities can function as proxies for the limits of grammar, the ungrammatical condition will stand out among the conditions that involve linguistic violations, showing a spike in the surprisal rates. Our results do not reveal a unique surprisal signature for ungrammatical prompts: across models, we observe comparable surprisal for the pragmatically odd condition, and the semantically odd condition consistently shows even higher surprisal. We thus demonstrate that probabilities do not constitute reliable proxies for grammaticality, showing that probability differences alone cannot be used to infer model-internal representations of syntactic knowledge. Consequently, claims about models being able to distinguish possible from impossible language need verification through a different methodology.
Abstract:A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to the models sensitivity to what crosses the limits of grammatically impossible language, this evidence has been contested on the grounds of the soundness of the testing material. We use model-internal representations as a direct means of investigating the way Large Language Models represent the grammatical-ungrammatical distinction. In a novel benchmark, we elicit sum log probabilities from 8 models and compute minimal-pair surprisal differences, juxtaposing probabilities assigned to grammatical sentences to probabilities assigned to (i) lower frequency grammatical sentences, (ii) ungrammatical sentences, (iii) semantically odd sentences, and (iv) pragmatically odd sentences. The prediction is that if string-probabilities can function as proxies for the limits of grammar, the ungrammatical condition will stand out among the conditions that involve linguistic violations, showing a spike in the surprisal rates. Our results do not reveal a unique surprisal signature for ungrammatical prompts: across models, we observe comparable surprisal for the pragmatically odd condition, and the semantically odd condition consistently shows even higher surprisal. We thus demonstrate that probabilities do not constitute reliable proxies for grammaticality, showing that probability differences alone cannot be used to infer model-internal representations of syntactic knowledge. Consequently, claims about models being able to distinguish possible from impossible language need verification through a different methodology.
Computation and Language (cs.CL)
arXiv:2509.15114 [cs.CL]
(or arXiv:2509.15114v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2509.15114 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 317 words
Related coverage
Quantum State Analysis Achieves Unprecedented Accuracy with Fewer Measurements
Quantum Zeitgeist · Jun 10, 2026
A unified framework for anomalous boson bunching
arXiv · Jul 23, 2026
An Operator-Norm Approach to Security with Quantum Advice
arXiv · Sep 30, 2026
On the Token Value Inequality in Efficient Reasoning
arXiv · Sep 30, 2026