aiqbench analysis
This vendor-run benchmark finds that the data layer around an LLM moved Text-to-SQL accuracy far more than the choice of model: a semantic layer added 20 to 31 points across all seven models, while swapping models moved it by up to 17.6. Tool-call habits, such as how often a model tests queries before submitting, did not predict scores, so readers should weigh process metrics cautiously. Because AtScale sells a semantic layer and ran the study, the results need independent replication before being treated as general guidance.
A better LLM isn’t the biggest upgrade for Text-to-SQL
Executive Summary Seven LLMs answered the same 410 business questions from the BIRD-Interact benchmark, once against PostgreSQL with full documentation and once through an AtScale semantic layer. Their tool-call logs show seven working styles. Grok […] The post A better LLM isn’t the biggest upgrade for Text-to-SQL appeared first on AtScale.
Seven LLMs answered the same 410 business questions from the BIRD-Interact benchmark, once against PostgreSQL with full documentation and once through an AtScale semantic layer. Their tool-call logs show seven working styles. Grok 4.3 submits SQL more often than it tests. Kimi K2.5 tests six times a task. None of those habits predicted the score. Switching LLMs moved accuracy by up to 17.6 points. The semantic layer added 20 to 31 points to every one of them. Every LLM worked the problem its own way. Grok 4.3 submits SQL 4.5 times a task and runs a test 0.7 times. Kimi K2.5 tests six times a task. Gemini 3.1 Pro asks the user most, at 2.2 clarifying questions per task. Those habits did not predict the score. Kimi ran out of budget on 81% of tasks and Gemini on 75%, and they finished 12.9 points apart. The semantic layer mattered more than the LLM. Switching LLMs moved accuracy by up to 17.6 points. The semantic layer added 20 to 31 points to every one of them, including the LLM already in the lead. An open-weight LLM kept pace with a frontier LLM. Kimi K2.5 scored 49.3% against GPT-5’s 50.5%, at a…
Excerpt shown under the publisher's RSS terms · 200 words · full text and figures are on the publisher's site