aiqbench analysis
This study provides a rigorous, preregistered comparison between single-round Vector RAG and LLM-compiled wikis for research corpus retrieval, revealing that neither architecture dominates across all metrics of synthesis cost, citation support, or holistic groundedness. The results highlight significant variability in evaluation outcomes depending on the scoring granularity used by judges and suggest that decomposition-retrieval variants can mitigate some token costs associated with wiki-based approaches without sacrificing claim-level evidence. Ultimately, researchers must carefully select retrieval strategies based on specific needs for organization versus atomic fact-checking rather than assuming a universal architectural winner exists.
Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus
arXiv:2605.18490v2 Announce Type: replace Abstract: We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: a single-round Vector RAG system and an LLM-compiled markdown wiki browsed by a tool-using agent. Both systems answered the same 13 questions over 24 papers using the same answer-generating model, and their answers were scored by two blinded LLM judges. The three preregistered predictions, in registered order, came out one weakly supported, one supported, and one refuted. The wiki was predicted to synthesize better across papers; it scored much better at connecting findings, but its organization advantage fell below the registered threshold once both judges' scores were combined. RAG was predicted to hold its own on single-fact lookup, and it met the registered test, though the second judge alone would have refuted it. The wiki was predicted to be expensive to build and cheap to query; the build side held by roughly two orders of magnitude, but the query side reversed: the wiki spent about 21 times more tokens per query, so no break-even point exists. Two exploratory analyses explain the disagreement. A decomposition-retrieval variant of RAG removes almost all of the wiki's synthesis advantage at lower token cost, though not its advantage in claim-by-claim citation support. Holistic groundedness scoring disagrees with atomized citation checking by direction, and between judges: rank agreement on that criterion is near zero (rho = 0.04), against rho = 0.81 on the most concretely defined criterion. Grounded research synthesis is therefore not a single capability: systems differ in how well they organize evidence, how well their citations support each claim, and what they cost to run, and no architecture here was best on all three. Which one appears to win depends on the retrieval baseline, the scoring granularity, and the judge.
Abstract:We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: a single-round Vector RAG system and an LLM-compiled markdown wiki browsed by a tool-using agent. Both systems answered the same 13 questions over 24 papers using the same answer-generating model, and their answers were scored by two blinded LLM judges. The three preregistered predictions, in registered order, came out one weakly supported, one supported, and one refuted. The wiki was predicted to synthesize better across papers; it scored much better at connecting findings, but its organization advantage fell below the registered threshold once both judges' scores were combined. RAG was predicted to hold its own on single-fact lookup, and it met the registered test, though the second judge alone would have refuted it. The wiki was predicted to be expensive to build and cheap to query; the build side held by roughly two orders of magnitude, but the query side reversed: the wiki spent about 21 times more tokens per query, so no break-even point exists. Two exploratory analyses explain the disagreement. A decomposition-retrieval variant of RAG removes almost all of the wiki's synthesis advantage at lower token cost, though not its advantage in claim-by-claim citation support. Holistic groundedness scoring disagrees with atomized citation checking by direction, and between judges: rank agreement on that criterion is near zero (rho = 0.04), against rho = 0.81 on the most concretely defined criterion. Grounded research synthesis is therefore not a single capability: systems differ in how well they organize evidence, how well their citations support each claim, and what they cost to run, and no architecture here was best on all three. Which one appears to win depends on the retrieval baseline, the scoring granularity, and the judge.
v2: two-judge reanalysis of the decomposition-RAG ablation (groundedness advantage +1.15 to +0.15); H3a adjudicated; one registered-plan deviation disclosed; artifact deposit at this http URL title corrected
Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2605.18490 [cs.CL]
(or arXiv:2605.18490v2 [cs.CL] for this version)
Full text under the publisher's open licence · 335 words
Related coverage
Iteratively Tuning hipBLASLt TensileLite Kernels: A Smaller Search, a Faster Kernel
AMD · Sep 9, 2026
Lattice-Based Cryptography Explained
Quantum Zeitgeist · Aug 15, 2026
Performance Profiling on AMD GPUs – Part 5: Profiling-Driven Kernel Optimization with an AI Code-Assist Tool
AMD · Jul 16, 2026
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Databricks · Oct 5, 2026