analiza aiqbench
This research argues that language should function as a boundary-layer compressor within multimodal models rather than an internal representation, drawing parallels to how human cognition separates linguistic indexing from sensory processing. Experiments on vision-language models reveal that embedding text internally degrades reliability during cue conflicts and reduces conceptual diversity, suggesting a structural shift toward externalizing the codebook is critical for auditability and robustness.
Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition
arXiv:2609.07474v1 Announce Type: new Abstract: Language models compute over tokens: language is their input, their output, and increasingly their internal representation. Whether language should keep all of these positions depends on what language does to the system that uses it. The one system with a century of data on that question is the human. We review what language does to human perception, the brain, and thought, and read the same evidence against multimodal models and language models. Throughout, we treat language as a compressor that runs on a shared codebook: a word is an index, the content is in the receiver, and a community maintains the codebook. In humans the compression is measurable, learning the codebook reorganizes the senses, and thought survives the loss of language. We then measure the rule that models apply when two cues disagree, with cue-conflict experiments on six vision-language models and two robot policies. Surviving cues are weighted in the order their reliabilities prescribe, at 11 to 82\% of the ideal observer's slope, and many answers copy the text. One policy family drops a cue that adds no information beyond the others rather than down-weighting it, another keeps it at a weight that fails when the cues conflict, and a visual cue that identifies the task in every training frame is never learned, because the language pathway already fits the data. Language models are the best current models of the human language network, and they have entered the human speech community, shifting word frequencies while alignment narrows their conceptual diversity. We close with seven implications for token-based systems. Language belongs at a model's boundary and in the shared codebook, as in the brain, not as its internal representation; the price of leaving the codebook inside is auditability.
Title:Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition
Abstract:Language models compute over tokens: language is their input, their output, and increasingly their internal representation. Whether language should keep all of these positions depends on what language does to the system that uses it. The one system with a century of data on that question is the human. We review what language does to human perception, the brain, and thought, and read the same evidence against multimodal models and language models. Throughout, we treat language as a compressor that runs on a shared codebook: a word is an index, the content is in the receiver, and a community maintains the codebook. In humans the compression is measurable, learning the codebook reorganizes the senses, and thought survives the loss of language. We then measure the rule that models apply when two cues disagree, with cue-conflict experiments on six vision-language models and two robot policies. Surviving cues are weighted in the order their reliabilities prescribe, at 11 to 82\% of the ideal observer's slope, and many answers copy the text. One policy family drops a cue that adds no information beyond the others rather than down-weighting it, another keeps it at a weight that fails when the cues conflict, and a visual cue that identifies the task in every training frame is never learned, because the language pathway already fits the data. Language models are the best current models of the human language network, and they have entered the human speech community, shifting word frequencies while alignment narrows their conceptual diversity. We close with seven implications for token-based systems. Language belongs at a model's boundary and in the shared codebook, as in the brain, not as its internal representation; the price of leaving the codebook inside is auditability.
Computation and Language (cs.CL)
arXiv:2609.07474 [cs.CL]
(or arXiv:2609.07474v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.07474 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 400 cuvinte
Subiecte conexe
Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception
arXiv · 29 oct. 2026
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
arXiv · 21 iul. 2026
SomaAlign: Reliability-Aware Multimodal Representation Learning for Soft Robotic Grasping
arXiv · 3 nov. 2026
The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?
arXiv · 5 oct. 2026