aiqbench analysis
This study investigates whether progressive disclosure strategies reduce computational costs for LLM agents maintaining large knowledge bases without sacrificing answer quality. The results indicate that while cost savings are consistent across conditions, the anticipated efficiency gains vanish for capable agents that can infer page locations directly, and answer quality is not guaranteed when forcing catalog loading or under stricter evaluation criteria.
Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation
arXiv:2607.04576v2 Announce Type: replace Abstract: LLM agents now often answer questions from knowledge bases they help maintain. A common intuition says progressive disclosure should make this cheaper. Instead of loading one large index, the agent reads a compact catalog and one-line page summaries, then opens only the pages it needs. We tested that intuition in a preregistered study on a real 709-page markdown knowledge base maintained by an LLM. We retrofitted it for progressive disclosure and built four versions that differ only in how the agent reaches the pages. The pages themselves are identical in every version, so any difference comes from the access structure alone. Each version was tested three ways, with the agent following a set protocol, choosing its own path, or made to load the catalog first. A judge from a different model family graded the answers blind against verified reference answers. A preparatory pilot changed the question. A capable agent never loaded the large index at all. It worked out from the question where a page was and read it directly. The saving we set out to measure did not exist for such an agent, so we made answer quality the primary outcome. Quality held. Answers from the retrofitted knowledge base were as good as answers from the original, within a margin we set in advance. Two limits apply. Our human rater and the model judge agreed far less than the plan required, so the quality result rests on the judge, backed by sensitivity checks. Quality was also not shown to hold when the agent was forced to load the catalog first, or on the two most reliably graded criteria under a stricter test. Cost fell clearly in every condition we tested, and the retrofitted version cited fewer pages and took fewer tool turns per answer.
Abstract:LLM agents now often answer questions from knowledge bases they help maintain. A common intuition says progressive disclosure should make this cheaper. Instead of loading one large index, the agent reads a compact catalog and one-line page summaries, then opens only the pages it needs. We tested that intuition in a preregistered study on a real 709-page markdown knowledge base maintained by an LLM. We retrofitted it for progressive disclosure and built four versions that differ only in how the agent reaches the pages. The pages themselves are identical in every version, so any difference comes from the access structure alone. Each version was tested three ways, with the agent following a set protocol, choosing its own path, or made to load the catalog first. A judge from a different model family graded the answers blind against verified reference answers. A preparatory pilot changed the question. A capable agent never loaded the large index at all. It worked out from the question where a page was and read it directly. The saving we set out to measure did not exist for such an agent, so we made answer quality the primary outcome. Quality held. Answers from the retrofitted knowledge base were as good as answers from the original, within a margin we set in advance. Two limits apply. Our human rater and the model judge agreed far less than the plan required, so the quality result rests on the judge, backed by sensitivity checks. Quality was also not shown to hold when the agent was forced to load the catalog first, or on the two most reliably graded criteria under a stricter test. Cost fell clearly in every condition we tested, and the retrofitted version cited fewer pages and took fewer tool turns per answer.
15 pages, 3 figures, 6 tables. v2 states its two limits in the abstract. Our human rater and the model judge agreed far less than the plan required. Quality was not shown to hold when the agent had to load the catalog first, or on the two most reliably graded criteria. Preregistered on OSF at this https URL, DOI https://doi.org/10.17605/OSF.IO/FEKA7
Computation and Language (cs.CL); Computers and Society (cs.CY); Information Retrieval (cs.IR)
arXiv:2607.04576 [cs.CL]
(or arXiv:2607.04576v2 [cs.CL] for this version)
Full text under the publisher's open licence · 374 words
Related coverage
Scaling Karpathy’s LLM wiki: Why your knowledge base needs a graph
Neo4j · Aug 31, 2026
Import AI 472: DeepMind’s cheating math agents; populist AI policies; and Forethought theorizes a nightwatchman
Import AI · Sep 7, 2026
Presentation: Context Engineering at LinkedIn: How We Built an Organizational Context Layer for AI Agents with MCP
InfoQ · Sep 19, 2026
An Organizational Second Brain: Building an AI That Learns From Experts
Meta Engineering · Sep 2, 2026