aiqbench analysis
This research investigates whether off-the-shelf multimodal encoders like CLIP can replicate human gaze patterns in visual world experiments without fine-tuning or generative architectures. By applying attribution methods to a bi-modal model, the study demonstrates that standard language-vision models possess latent cognitive adequacy for predicting human attention during simultaneous visual and linguistic processing. This finding bridges a gap in computational psycholinguistics by showing that complex multimodal reasoning can emerge from pre-trained models applied to unimodal tasks.
Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments
arXiv:2608.07282v2 Announce Type: replace Abstract: A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.
Abstract:A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.
Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2608.07282 [cs.CL]
(or arXiv:2608.07282v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.07282 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
Dyna-2: General Purpose Physical Agents with Hour-Long Autonomy — Jason Ma, Dyna Robotics
Latent Space · Oct 13, 2026
Emergent Goal-Directed Attention in Large Vision-Language Models
arXiv · Sep 9, 2026
[AINews] Gemini 4 Argon: GDM’s answer to Astra/Fable, with 1M output
Latent Space · Oct 1, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026