analiza aiqbench
This paper addresses the critical deployment challenge where LLM-based text-to-SQL systems generate fluent but factually incorrect answers that cannot be easily inspected by end-users or agents. The authors propose "structural abstention," an architectural pattern separating a generative interface from a deterministic kernel to ensure unanswerable queries are refused rather than hallucinated, prioritizing system reliability over statistical confidence scores.
Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact
arXiv:2608.13926v1 Announce Type: cross Abstract: Large language models have made natural language interfaces to databases (NLIDB) newly credible, but LLM text-to-SQL systems fail in a way that matters for deployment: a hallucinated column or a mis-aggregated total yields a fluent wrong answer, indistinguishable at the point of use from a right one. Where the consumer cannot inspect the generated query, as in enterprise AI deployments and operational dashboards, and increasingly where the consumer is a tool-using agent rather than a person, accuracy alone is insufficient: nothing marks which answers to distrust. This is a reliability problem before it is an accuracy problem. We propose an architectural pattern for such systems, a trusted kernel with a generative shell, resting on one invariant: a component that can fabricate may influence which question the system answers, never which value it returns. A generative shell interprets underspecified input and phrases replies; a deterministic kernel matches fully specified questions against a bounded set of answerable question shapes and compiles them to queries by deterministic execution. The two meet at a confirmation the user reads before any value is computed, and requests the kernel cannot express are declined rather than approximated. We call this structural abstention, and distinguish it from the statistical abstention of selective prediction and calibrated confidence: refusal here needs no confidence estimate, because unanswerable requests are unrepresentable. We specify the pattern implementation-independently, give a five-decision recipe and work it across three domains, extend the invariant from returned values to the actions of agentic systems, and report a two-year production case study alongside two generative alternatives, a fine-tuned parser and a tool-retrieval agent. We close against enterprise and reliability benchmarks published since.
Abstract:Large language models have made natural language interfaces to databases (NLIDB) newly credible, but LLM text-to-SQL systems fail in a way that matters for deployment: a hallucinated column or a mis-aggregated total yields a fluent wrong answer, indistinguishable at the point of use from a right one. Where the consumer cannot inspect the generated query, as in enterprise AI deployments and operational dashboards, and increasingly where the consumer is a tool-using agent rather than a person, accuracy alone is insufficient: nothing marks which answers to distrust. This is a reliability problem before it is an accuracy problem. We propose an architectural pattern for such systems, a trusted kernel with a generative shell, resting on one invariant: a component that can fabricate may influence which question the system answers, never which value it returns. A generative shell interprets underspecified input and phrases replies; a deterministic kernel matches fully specified questions against a bounded set of answerable question shapes and compiles them to queries by deterministic execution. The two meet at a confirmation the user reads before any value is computed, and requests the kernel cannot express are declined rather than approximated. We call this structural abstention, and distinguish it from the statistical abstention of selective prediction and calibrated confidence: refusal here needs no confidence estimate, because unanswerable requests are unrepresentable. We specify the pattern implementation-independently, give a five-decision recipe and work it across three domains, extend the invariant from returned values to the actions of agentic systems, and report a two-year production case study alongside two generative alternatives, a fine-tuned parser and a tool-retrieval agent. We close against enterprise and reliability benchmarks published since.
26 pages, 5 figures, 5 tables. Technical report. Describes architecture and design principles only; contains no code, schemas, datasets, or performance metrics
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Databases (cs.DB)
H.2.3; H.5.2; I.2.7
arXiv:2608.13926 [cs.AI]
(or arXiv:2608.13926v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.13926 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 322 cuvinte
Subiecte conexe
Lattice-Based Cryptography Explained
Quantum Zeitgeist · 15 aug. 2026
The Entangled Dimensions of Decision Theory
LessWrong · 30 iul. 2026
The payload: the semantic layer, memory, and the loop that fills it
Neo4j · 10 aug. 2026
How GoFundMe became America’s backup plan
The Verge · 24 aug. 2026