aiqbench analysis
This study challenges the necessity of LLM-based fact extraction for conversational memory, demonstrating that a typed decision model can select raw dialogue turns with non-inferior performance at significantly lower computational cost. The findings resolve conflicting literature by showing that while ranking provides substantial gains under tight context budgets, its marginal utility diminishes as available context expands, making simple selection viable for many applications.
When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model
arXiv:2609.34227v1 Announce Type: cross Abstract: Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.
Abstract:Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.
21 pages, 9 figures. Pre-registered: plan doi:https://doi.org/10.5281/zenodo.22970745, amendment doi:https://doi.org/10.5281/zenodo.22977848. Preprint also at doi:https://doi.org/10.5281/zenodo.22985242. Code and data: this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2609.34227 [cs.AI]
(or arXiv:2609.34227v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.34227 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 245 words
Related coverage
[AINews] not much happened today
Latent Space · Oct 3, 2026
Extending public sector intelligence with Agentforce and AWS
AWS · Sep 22, 2026
Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore
AWS · Oct 16, 2026
GEAK V3: Agent-Driven, Repository-Level GPU Kernel Optimization across HIP, Triton, and FlyDSL on AMD GPUs
AMD · Jul 20, 2026