analiza aiqbench
This paper is squarely about LLM behavior, so it passes the topic gate. It introduces VagueBench, a benchmark for when and what clarifying questions a model should ask, and DialogGRPO, an offline RL method for training that behavior. The headline finding, that fewer than 11% of underspecified queries trigger a follow-up across seven models, is a useful diagnostic, though the reported gains rest on the authors' own synthetic degradation of queries, so real-world transfer remains to be seen.
Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations
arXiv:2611.01878v1 Announce Type: new Abstract: When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up question. To benchmark and train this capability, we introduce VagueBench, where each seed query is annotated with structured intent and detail units then degraded into intent-vague and fully-vague variants via controlled removal of linguistic cues, yielding known hidden information sets for evaluating both when and what to ask. We propose DialogGRPO, an offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph, where candidate questions at each node receive dense turn-level rewards combining immediate information gain with lookahead value through group-relative comparison, enabling fine-grained credit assignment without runtime simulation. DialogGRPO elicits missing information on over 88% of vague queries with substantial gains in both recovery efficiency and answer coverage, without over-asking on complete queries, outperforms prompting and test-time-scaling baselines, and generalizes zero-shot to MultiWOZ 2.4 and AmbigQA.
Citește articolul complet pe arXivSubiecte conexe
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
arXiv · 10 iun. 2026
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
arXiv · 7 aug. 2026
Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errors
arXiv · 2 oct. 2026
Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
arXiv · 5 iun. 2026