aiqbench analysis
This arXiv paper evaluates LLM hallucinations in Text-to-SQL by decomposing queries into sub-queries and tracking whether errors recur across the generation path or first appear in the final query. Its main contribution is a trajectory-level evaluation method, which suggests that final-output benchmarks can miss where errors originate. Results on five LLMs and an expanded BIRD-mini benchmark are preliminary, so readers should treat the reported probabilities as indicative rather than settled.
Tracing Recurrent and Emergent Hallucination Patterns in Decomposed Text-to-SQL Generation
arXiv:2610.27720v1 Announce Type: new Abstract: Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors. Most evaluations characterize such failures only in the final generated query, making it difficult to determine whether a related error was also observed in an earlier, simpler step or appears only at the final stage. We present a drill-down evaluation framework that decomposes Text-to-SQL queries into progressively more complex executable sub-queries and corresponding natural-language questions, allowing rule-based hallucination categories to be examined across a generation path. We distinguish recurrent hallucinations, in which the same rule-based category is observed in an earlier step and again in the final query, from emergent hallucinations, in which the category is observed at the final step without an earlier detection in the same path. Across five LLMs evaluated on an expanded BIRD-mini benchmark, several categories show appreciable recurrence probabilities, while clause abuse shows comparatively higher conditional emergence. Under the conversational-history condition, recurrence probabilities are generally higher and emergence probabilities are generally lower. These observations indicate that the two conditional measures do not move uniformly in the same direction under additional context and motivate trajectory-level analysis in addition to final-output evaluation. Code and evaluation artifacts are available in the project repository. https://github.com/HaydenMM/bird-mini-drilldown
Abstract:Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors. Most evaluations characterize such failures only in the final generated query, making it difficult to determine whether a related error was also observed in an earlier, simpler step or appears only at the final stage. We present a drill-down evaluation framework that decomposes Text-to-SQL queries into progressively more complex executable sub-queries and corresponding natural-language questions, allowing rule-based hallucination categories to be examined across a generation path. We distinguish recurrent hallucinations, in which the same rule-based category is observed in an earlier step and again in the final query, from emergent hallucinations, in which the category is observed at the final step without an earlier detection in the same path. Across five LLMs evaluated on an expanded BIRD-mini benchmark, several categories show appreciable recurrence probabilities, while clause abuse shows comparatively higher conditional emergence. Under the conversational-history condition, recurrence probabilities are generally higher and emergence probabilities are generally lower. These observations indicate that the two conditional measures do not move uniformly in the same direction under additional context and motivate trajectory-level analysis in addition to final-output evaluation. Code and evaluation artifacts are available in the project repository. this https URL
Computation and Language (cs.CL)
arXiv:2610.27720 [cs.CL]
(or arXiv:2610.27720v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.27720 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 299 words
Related coverage
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
arXiv · Sep 30, 2026
RadioMaster: Multi-Agent System for Autonomous Radio Signal Generation
arXiv · Jul 29, 2026
Redesigning and Auditing Deep Research Writing for Faithful Reports
arXiv · Sep 1, 2026
GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification
arXiv · Oct 6, 2026