aiqbench analysis
This study identifies a novel metric, trajectory extrapolation error, which independently predicts human reading times by capturing the local momentum of language model hidden states, distinct from traditional surprisal measures. The findings demonstrate that processing costs are driven by both prediction error and the direction of interpretive evolution, with effects scaling with model size and persisting across different architectural encodings. For researchers in cognitive modeling and interpretability, this suggests that analyzing the dynamic trajectory of hidden states is essential for a complete understanding of sequential language processing.
Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal
arXiv:2606.05346v1 Announce Type: new Abstract: Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).
Abstract:Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).
Computation and Language (cs.CL)
arXiv:2606.05346 [cs.CL]
(or arXiv:2606.05346v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.05346 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 327 words
Related coverage
Pivot-Centric Trajectory Prediction: Bridging Long Horizons via Dynamical Guidance
arXiv · Aug 5, 2026
All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents
arXiv · Oct 17, 2026
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
arXiv · Sep 9, 2026
Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
arXiv · Jun 9, 2026