analiza aiqbench
The paper evaluates how LLM-based financial assistants handle underspecified queries, such as ambiguous data labels, unstated product variants, and missing investor or tax details, using Indian mutual funds as a testbed where every correct answer can be computed. Its main finding is that answers can match their source data and still answer a different question, so standard data-grounded checks miss the error. For AI readers, the work points to a concrete evaluation gap: reliability depends on how a query's implicit assumptions are resolved, and AI judges need explicit field semantics to catch silent misreadings. The paper proposes controls but reports mechanisms rather than failure rates.
Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries
arXiv:2610.38631v1 Announce Type: new Abstract: Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.
Abstract:Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.
Accepted for oral presentation at the 3rd Workshop on LLMs and Generative AI for Finance (AI4F) at ACM ICAIF '26, Milan, Italy, November 14, 2026. Non-archival. 16 pages, 4 figures
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); General Finance (q-fin.GN)
arXiv:2610.38631 [cs.AI]
(or arXiv:2610.38631v1 [cs.AI] for this version)
Text integral sub licența deschisă a publicației · 240 cuvinte
Subiecte conexe
AI Evaluation Should Require Standardized Item-Level Data Releases
arXiv · 20 oct. 2026
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
arXiv · 16 sept. 2026
How Specific Do I Need to Be? Effects of User Query Underspecification on Deep Research Agents
arXiv · 30 oct. 2026
FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
arXiv · 15 oct. 2026