aiqbench analysis
This paper addresses the critical scalability bottleneck of full-context injection in long LLM sessions by introducing "Pull," a method that lazily materializes only necessary conversation turns while preserving reversible access to history. By decoupling metadata management from costly LLM calls, the approach significantly reduces token consumption without sacrificing performance on complex multi-hop reasoning tasks. This represents a fundamental architectural shift toward efficient, stateful memory management for next-generation conversational agents.
Pull: Lazy Materialization of Working Memory for Stateful LLM Conversations
arXiv:2609.14773v1 Announce Type: new Abstract: As LLM conversations grow to hundreds of turns, full-context injection incurs $O(N^2)$ cumulative token costs, while lossy summarization or hard truncation irreversibly discards historical state. We propose Pull, a session router that maintains an addressable metadata directory via a local, deterministic Purifier (zero LLM calls, millisecond-level latency). At query time, the LLM lazily materializes only the turns it needs; unmaterialized turns remain accessible but collapsed. Unlike irreversible compression, Pull's materialization is reversible; subsequent queries can expand any collapsed turn. On LoCoEval (128 conversations, 12,780 turns), Pull reduces per-query context tokens (Phase 2) by 75.1 percent on single-hop tasks with equivalent quality ($\Delta = -0.002$, n.s.) and by 72.0 percent on multi-hop tasks with no quality loss ($\Delta = +0.017$). A controlled routing benchmark (7,831 queries x 10 methods) shows that entity lifecycle tracking is empirically a prerequisite for distance-independent routing. On BEAM 1M (14 conversations, 263 questions), Pull improves F1 by +55.2 percent over a truncation baseline.
Abstract:As LLM conversations grow to hundreds of turns, full-context injection incurs $O(N^2)$ cumulative token costs, while lossy summarization or hard truncation irreversibly discards historical state. We propose Pull, a session router that maintains an addressable metadata directory via a local, deterministic Purifier (zero LLM calls, millisecond-level latency). At query time, the LLM lazily materializes only the turns it needs; unmaterialized turns remain accessible but collapsed. Unlike irreversible compression, Pull's materialization is reversible; subsequent queries can expand any collapsed turn. On LoCoEval (128 conversations, 12,780 turns), Pull reduces per-query context tokens (Phase 2) by 75.1 percent on single-hop tasks with equivalent quality ($\Delta = -0.002$, n.s.) and by 72.0 percent on multi-hop tasks with no quality loss ($\Delta = +0.017$). A controlled routing benchmark (7,831 queries x 10 methods) shows that entity lifecycle tracking is empirically a prerequisite for distance-independent routing. On BEAM 1M (14 conversations, 263 questions), Pull improves F1 by +55.2 percent over a truncation baseline.
12 pages, 3 figures. An earlier version was publicly released on Zenodo (DOI: https://doi.org/10.5281/zenodo.21984705). Code and reproduction scripts: this https URL
Computation and Language (cs.CL)
arXiv:2609.14773 [cs.CL]
(or arXiv:2609.14773v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.14773 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 197 words
