Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

arXiv:2604.15210v2 Announce Type: replace Abstract: Humor is one of the few cognitive tasks where getting the reasoning right matters as much as getting the answer right. While recent work evaluates humor understanding on benchmarks such as the New Yorker Cartoon Caption Contest (NYCC), it largely treats it as black-box prediction, overlooking the structured reasoning processes underlying humor comprehension. We introduce IRS (Incongruity-Resolution Supervision), a framework that decomposes humor understanding into three components: Incongruity Modeling, which identifies mismatches in the visual scene; Resolution Modeling, which constructs coherent reinterpretations of these mismatches; and Preference Alignment, which evaluates candidate interpretations under human judgments. Grounded in incongruity-resolution theory and expert captionist practice, IRS supervises intermediate reasoning process through structured traces that make the path from visual perception to humorous interpretation explicit and learnable. Across 7B, 32B, and 72B models on NYCC, IRS improves performance across caption matching and ranking, with IRS-72B achieving the strongest model performance on ranking (76.10%), surpassing both non-expert human performance and all evaluated open- and closed-source multimodal baselines. Zero-shot transfer further shows that IRS learns generalizable reasoning patterns.

Abstract:Humor is one of the few cognitive tasks where getting the reasoning right matters as much as getting the answer right. While recent work evaluates humor understanding on benchmarks such as the New Yorker Cartoon Caption Contest (NYCC), it largely treats it as black-box prediction, overlooking the structured reasoning processes underlying humor comprehension. We introduce IRS (Incongruity-Resolution Supervision), a framework that decomposes humor understanding into three components: Incongruity Modeling, which identifies mismatches in the visual scene; Resolution Modeling, which constructs coherent reinterpretations of these mismatches; and Preference Alignment, which evaluates candidate interpretations under human judgments. Grounded in incongruity-resolution theory and expert captionist practice, IRS supervises intermediate reasoning process through structured traces that make the path from visual perception to humorous interpretation explicit and learnable. Across 7B, 32B, and 72B models on NYCC, IRS improves performance across caption matching and ranking, with IRS-72B achieving the strongest model performance on ranking (76.10%), surpassing both non-expert human performance and all evaluated open- and closed-source multimodal baselines. Zero-shot transfer further shows that IRS learns generalizable reasoning patterns.

Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2604.15210 [cs.AI]

(or arXiv:2604.15210v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2604.15210 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

Related coverage