aiqbench analysis
The Hummus dataset introduces a novel annotation scheme for humorous multimodal metaphors, revealing that current multimodal large language models (MLLMs) still struggle to effectively integrate visual and textual information for this specific task. This release highlights a significant gap in the reasoning capabilities of state-of-the-art models when processing incongruity-based humor, offering a targeted benchmark for future research in multimodal understanding.
Hummus: A Dataset of Humorous Multimodal Metaphor Use
arXiv:2504.02983v3 Announce Type: replace Abstract: Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.
Abstract:Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at this http URL.
Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2504.02983 [cs.CL]
(or arXiv:2504.02983v3 [cs.CL] for this version)
Full text under the publisher's open licence · 172 words
Related coverage
From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignment
arXiv · Aug 25, 2026
BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset
arXiv · Sep 9, 2026
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
arXiv · Aug 25, 2026
Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
arXiv · Jul 23, 2026