aiqbench analysis
This research introduces Candidate Set Visual Grounding (CSVG), a new task designed to evaluate how vision-language models manage ambiguity by tracking multiple plausible object referents rather than forcing immediate unique identification. The study reveals that current models struggle with maintaining complete candidate sets across dialogue turns due to premature commitment, though performance can be partially improved with specific instructions. This work advances the understanding of interactive visual grounding and context accumulation in multimodal AI systems.
Recognizing Dialogue Ambiguity via Candidate Set Visual Grounding
arXiv:2610.19723v1 Announce Type: new Abstract: Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects in a visual scene, it is necessary to maintain and update the set of plausible referents as context accumulates, without assuming that the available information identifies a unique object. However, prior work on interactive visual grounding, which localizes objects referred to by linguistic expressions, has focused on identifying a unique object once sufficient dialogue context becomes available. To address this gap, we propose Candidate Set Visual Grounding (CSVG), a task for estimating the set of objects consistent with the given dialogue context. We further introduce CandVG, a benchmark that evaluates how the candidate set changes across dialogue turns. Our evaluation of vision-language models shows that stronger visual grounding capability does not necessarily yield better CSVG performance. In particular, models have difficulty predicting the complete candidate set, owing to challenges in candidate generation and premature commitment to a small set of candidates in early turns. Nevertheless, we find that this limitation can be partially mitigated in the evaluated large-scale models using instructions designed for CSVG.
Abstract:Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects in a visual scene, it is necessary to maintain and update the set of plausible referents as context accumulates, without assuming that the available information identifies a unique object. However, prior work on interactive visual grounding, which localizes objects referred to by linguistic expressions, has focused on identifying a unique object once sufficient dialogue context becomes available. To address this gap, we propose Candidate Set Visual Grounding (CSVG), a task for estimating the set of objects consistent with the given dialogue context. We further introduce CandVG, a benchmark that evaluates how the candidate set changes across dialogue turns. Our evaluation of vision-language models shows that stronger visual grounding capability does not necessarily yield better CSVG performance. In particular, models have difficulty predicting the complete candidate set, owing to challenges in candidate generation and premature commitment to a small set of candidates in early turns. Nevertheless, we find that this limitation can be partially mitigated in the evaluated large-scale models using instructions designed for CSVG.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2610.19723 [cs.CL]
(or arXiv:2610.19723v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.19723 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
arXiv · Sep 22, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Cost-efficient Active Learning for Referring Image Segmentation and Grounding
arXiv · Sep 2, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026