analiza aiqbench
This paper challenges the uncritical application of surprisal metrics derived from large language models, arguing that architectural differences significantly impact probability calculations and thus theoretical validity. The authors contend that treating LLM outputs as interchangeable representations obscures critical algorithmic commitments required for rigorous computational-level analysis in psycholinguistics. Consequently, researchers must re-evaluate their methodologies before using these black-box probabilities to test linguistic theories without accounting for model-specific biases.
surprisal is Not a Theory
arXiv:2607.20208v1 Announce Type: new Abstract: Surprisal Theory is often characterized as a computational-level explanation per (Marr, 1982). We argue in this work that, even though a computational level narrative has been used to support "representation-agnostic research" within computational psycholinguistics, the movement toward black box systems embodied by large language models (LLMs) does not exempt modelers using the surprisal metric from the representational decisions required by computational-level characterizations. In fact, we argue that the uncritical use of LLM-surprisal obfuscates the representational and algorithmic-level commitments of different models. In three analyses, we show that the choice of algorithm and model architecture play significant roles in the computation of language model probabilities. We advise that researchers who wish to test Surprisal Theory re-evaluate the practice of treating large language model probabilities as interchangeable
Abstract:Surprisal Theory is often characterized as a computational-level explanation per (Marr, 1982). We argue in this work that, even though a computational level narrative has been used to support "representation-agnostic research" within computational psycholinguistics, the movement toward black box systems embodied by large language models (LLMs) does not exempt modelers using the surprisal metric from the representational decisions required by computational-level characterizations. In fact, we argue that the uncritical use of LLM-surprisal obfuscates the representational and algorithmic-level commitments of different models. In three analyses, we show that the choice of algorithm and model architecture play significant roles in the computation of language model probabilities. We advise that researchers who wish to test Surprisal Theory re-evaluate the practice of treating large language model probabilities as interchangeable
Computation and Language (cs.CL)
arXiv:2607.20208 [cs.CL]
(or arXiv:2607.20208v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.20208 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 220 cuvinte
Subiecte conexe
Estimating Surprisal from the Speech Signal
arXiv · 14 oct. 2026
Surprisal Theory is Tautological (without Rational Grounding)
arXiv · 25 iul. 2026
III. Anthropic reasoning has issues with infinite worlds; D-SIA can fix this
LessWrong · 3 aug. 2026
Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
LessWrong · 27 iul. 2026