aiqbench analysis
This research identifies a significant "multimodal gap" where foundation models generate far less semantic variety in images than in text when presented with ambiguous words, revealing limitations in how meaning transfers across modalities. The study highlights that while LLMs can predict diverse interpretations of polysemous terms like 'bank', their actual image generation capabilities collapse into fewer senses compared to human imagination. These findings are critical for developers aiming to build more faithful and context-aware multimodal systems where visual outputs accurately reflect the full spectrum of linguistic nuance.
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
arXiv:2608.00410v1 Announce Type: new Abstract: Human language is highly polysemous. Many common words (e.g., 'bank' or 'palm') carry several distinct meanings that shape what humans communicate and imagine. Large language models (LLMs) have been shown to understand this multiplicity of meaning, but much less is known about how polysemy surfaces in other modalities such as images. We study this across 17 text-to-image and 15 text-generation models by giving each a polysemous word with no context to fix its meaning and measuring which senses are produced over many samples. We find a clear multimodal gap, where within every model family, generated images settle on far fewer senses than generated sentences (normalized entropy 0.10 vs. 0.25), and both are far less varied than what people imagine for the same words (normalized entropy 0.47). However, when we instead ask a model to list how often it would generate outputs corresponding to each possible meaning of a word, it predicts distributions that are more diverse than the actual space of outputs. These results reveal a multimodal gap in how foundation models express meaning, and how their understanding may not transfer faithfully nor equally across modalities.
Title:Where did the ambiguity go? Examining how multimodal models interpret polysemous words
Abstract:Human language is highly polysemous. Many common words (e.g., 'bank' or 'palm') carry several distinct meanings that shape what humans communicate and imagine. Large language models (LLMs) have been shown to understand this multiplicity of meaning, but much less is known about how polysemy surfaces in other modalities such as images. We study this across 17 text-to-image and 15 text-generation models by giving each a polysemous word with no context to fix its meaning and measuring which senses are produced over many samples. We find a clear multimodal gap, where within every model family, generated images settle on far fewer senses than generated sentences (normalized entropy 0.10 vs. 0.25), and both are far less varied than what people imagine for the same words (normalized entropy 0.47). However, when we instead ask a model to list how often it would generate outputs corresponding to each possible meaning of a word, it predicts distributions that are more diverse than the actual space of outputs. These results reveal a multimodal gap in how foundation models express meaning, and how their understanding may not transfer faithfully nor equally across modalities.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2608.00410 [cs.AI]
(or arXiv:2608.00410v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.00410 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 302 words
Related coverage
Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
arXiv · Sep 14, 2026
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · Jul 30, 2026
YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perception
arXiv · Sep 7, 2026
Eye movements separate prediction from structural difficulty
Quantum Zeitgeist · Aug 11, 2026