aiqbench analysis
This research addresses the tendency of large language models to prematurely answer ambiguous references instead of engaging in necessary clarification, proposing a lightweight method called BACE to optimize question-asking strategies. By constructing a controlled test environment from the SIMMC 2.1 dataset, the authors demonstrate that their approach consistently selects the optimal number of clarifying questions more effectively than existing post-training methods like GRPO and ArCHer. The work advances the understanding of collaborative dialogue dynamics and improves the reliability of LLMs in multi-turn conversational contexts where disambiguation is critical.
When to Ask, When to Answer: Strategic Clarification for Ambiguous Reference Resolution in Dialogue
arXiv:2611.05517v1 Announce Type: cross Abstract: For humans, referential disambiguation is a collaborative process involving clarification questions asked when they will meaningfully change the outcome. However, large language models (LLMs) have been shown to be biased toward premature answering, preferring definite single-turn responses over more clarification exchanges. In this paper, we study the problem of "strategic clarification" -- asking clarification questions in a reference game when their answers will narrow the set of candidate referents. We construct a controlled test environment from the SIMMC 2.1 dataset that records how many questions each ambiguous reference needs to be successfully resolved, and examine how baseline LLM behavior and common post-training methods ask clarifying questions vs. attempt to answer the referent. We also propose BACE (Belief-Aware Critic with Epistemic tracking), a lightweight action chooser method on top of a frozen LM backbone, and show that BACE more consistently chooses the optimal number of clarifying questions than methods such as GRPO and ArCHer.
Abstract:For humans, referential disambiguation is a collaborative process involving clarification questions asked when they will meaningfully change the outcome. However, large language models (LLMs) have been shown to be biased toward premature answering, preferring definite single-turn responses over more clarification exchanges. In this paper, we study the problem of "strategic clarification" -- asking clarification questions in a reference game when their answers will narrow the set of candidate referents. We construct a controlled test environment from the SIMMC 2.1 dataset that records how many questions each ambiguous reference needs to be successfully resolved, and examine how baseline LLM behavior and common post-training methods ask clarifying questions vs. attempt to answer the referent. We also propose BACE (Belief-Aware Critic with Epistemic tracking), a lightweight action chooser method on top of a frozen LM backbone, and show that BACE more consistently chooses the optimal number of clarifying questions than methods such as GRPO and ArCHer.
22 pages, 10 figures. Code and data: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2611.05517 [cs.CL]
(or arXiv:2611.05517v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.05517 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Uncertainty-Aware Clarification in LLM Agents with Information Gain
arXiv · Jun 3, 2026
One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
arXiv · Jul 25, 2026
We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation
arXiv · Sep 9, 2026
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
arXiv · Sep 1, 2026