aiqbench analysis
This research addresses a critical bottleneck in LLM-based text-to-SQL applications by systematically analyzing error patterns and introducing MapleDoctor, a framework that significantly improves query repair accuracy while reducing computational latency. The study moves beyond generic model capabilities to offer concrete engineering solutions for enhancing the reliability of AI-driven database interactions through rigorous detection and correction mechanisms.
Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors
arXiv:2501.09310v3 Announce Type: replace Abstract: Large language models (LLMs) have been adopted for text-to-SQL tasks, utilizing their in-context learning (ICL) capability to translate natural language questions into SQL queries. However, such a technique faces correctness problems. In this paper, we conduct the first comprehensive study of text-to-SQL errors of ICL-based techniques. Our study covers four representative ICL-based techniques, five basic repairing methods, two benchmarks, and two LLM settings. We find that text-to-SQL errors are widespread and summarize 27 error types of 7 categories. We also find that existing repairing attempts have limited correctness improvement while having high computational overhead and many mis-repairs. Based on these findings, we propose MapleDoctor, a novel text-to-SQL error detection and repairing framework. The evaluation demonstrates that MapleDoctor outperforms existing solutions by repairing 13.8% more queries with a negligible number of mis-repairs and reducing 67.4% repair latency. The artifact is publicly available at GitHub.
Abstract:Large language models (LLMs) have been adopted for text-to-SQL tasks, utilizing their in-context learning (ICL) capability to translate natural language questions into SQL queries. However, such a technique faces correctness problems. In this paper, we conduct the first comprehensive study of text-to-SQL errors of ICL-based techniques. Our study covers four representative ICL-based techniques, five basic repairing methods, two benchmarks, and two LLM settings. We find that text-to-SQL errors are widespread and summarize 27 error types of 7 categories. We also find that existing repairing attempts have limited correctness improvement while having high computational overhead and many mis-repairs. Based on these findings, we propose MapleDoctor, a novel text-to-SQL error detection and repairing framework. The evaluation demonstrates that MapleDoctor outperforms existing solutions by repairing 13.8% more queries with a negligible number of mis-repairs and reducing 67.4% repair latency. The artifact is publicly available at GitHub.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
arXiv:2501.09310 [cs.CL]
(or arXiv:2501.09310v3 [cs.CL] for this version)
Full text under the publisher's open licence · 161 words
Related coverage
Presentation: Beyond Prompting: Context Engineering and Memory Management for AI Systems at Scale
InfoQ · Jun 10, 2026
How mobility gives language models a deeper understanding of place
Google Research · Aug 21, 2026
NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI
NVIDIA · May 21, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026