analiza aiqbench
The article introduces CREATE, a new benchmark designed to evaluate the lateral reasoning and associative capabilities of vision language models on visual puzzles. By demonstrating that current VLMs struggle with these tasks compared to humans, the study highlights critical limitations in how AI handles creative problem-solving beyond simple text cues. This research is essential for understanding the boundaries of generative model creativity and guiding future improvements in multimodal reasoning.
CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test
arXiv:2610.18569v1 Announce Type: new Abstract: Vision language models (VLMs) are used in creative tasks such as visual storytelling, design ideation, art interpretation, and advertising. Yet we lack principled ways to measure the lateral reasoning of these models compared to human potential. We introduce CREATE, a semi-automatically constructed benchmark containing 155 visual Remote Associates Test problems. These problems require a solver to find an underlying connection among three seemingly dissimilar images. We conduct a human study (N=25) to validate and ground the quality of our benchmark, and evaluate 11 VLMs. Our findings show that most VLMs underperform humans on this task. Our experiments indicate that the associative ability of VLMs depends on the cue concepts provided in the text and does not universally extend to visual puzzles.
Abstract:Vision language models (VLMs) are used in creative tasks such as visual storytelling, design ideation, art interpretation, and advertising. Yet we lack principled ways to measure the lateral reasoning of these models compared to human potential. We introduce CREATE, a semi-automatically constructed benchmark containing 155 visual Remote Associates Test problems. These problems require a solver to find an underlying connection among three seemingly dissimilar images. We conduct a human study (N=25) to validate and ground the quality of our benchmark, and evaluate 11 VLMs. Our findings show that most VLMs underperform humans on this task. Our experiments indicate that the associative ability of VLMs depends on the cue concepts provided in the text and does not universally extend to visual puzzles.
Computation and Language (cs.CL)
arXiv:2610.18569 [cs.CL]
(or arXiv:2610.18569v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.18569 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 216 cuvinte
