analiza aiqbench
This research introduces LSREP, a rigorous evaluation protocol for conversational memory systems that exposes critical failure modes in local-first architectures like ICE v2 when compared to standard vector-RAG approaches. The study highlights significant performance gaps on dense datasets and temporal tasks, demonstrating that architectural efficiency gains often come at the cost of reliability without comprehensive longitudinal testing. These findings are essential for developers building persistent AI agents who need to understand the trade-offs between context budgeting, retrieval fidelity, and long-term memory retention.
LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture
arXiv:2609.16730v1 Announce Type: new Abstract: Conversational memory changes during use, so endpoint question answering alone cannot establish how a persistent state accumulates, ages, or incorporates revisions. We introduce LSREP, a Longitudinal State-Replay Evaluation Protocol combining ordered replay, explicit lifecycle schedules, repeated probes, evolving reference answers, and mechanism-fidelity checks. Its architectural case study is ICE v2, a local-first memory middleware with typed stores, retrieval fusion, and dynamic context budgets. The private, single-user instantiation contains 1,985 turns, 219 distinct probes, and 1,211 probe-checkpoint observations across 52 checkpoints. On three ordinary-density datasets, ICE v2 has a near-zero mean quality difference from vector-RAG while selecting 32% fewer fragments but using 6.6% more estimated prompt tokens. A fourth, dense dataset exposes catastrophic failures of the unbudgeted baseline. The fidelity audit limits attribution: procedural retrieval is defective, several mechanisms are unexercised, and graph utility is not established. In a complementary matched public diagnostic, ICE v2 loses decisively to pure vector-RAG on LongMemEval: 50.8% versus 72.8% in the evidence-only oracle and 43.0% versus 69.5% in full-S. Paired differences are -22.0 points (95% CI [-26.6, -17.4]) and -26.5 ([-31.3, -21.8]). Conservative abstention accompanies severe multi-session and temporal failures. ICE uses less context in this diagnostic, establishing a quality-cost trade-off rather than superior efficiency. Together, replay, fidelity auditing, and public endpoint testing expose distinct failure modes that neither architectural descriptions nor aggregate scores identify alone.
Abstract:Conversational memory changes during use, so endpoint question answering alone cannot establish how a persistent state accumulates, ages, or incorporates revisions. We introduce LSREP, a Longitudinal State-Replay Evaluation Protocol combining ordered replay, explicit lifecycle schedules, repeated probes, evolving reference answers, and mechanism-fidelity checks. Its architectural case study is ICE v2, a local-first memory middleware with typed stores, retrieval fusion, and dynamic context budgets. The private, single-user instantiation contains 1,985 turns, 219 distinct probes, and 1,211 probe-checkpoint observations across 52 checkpoints. On three ordinary-density datasets, ICE v2 has a near-zero mean quality difference from vector-RAG while selecting 32% fewer fragments but using 6.6% more estimated prompt tokens. A fourth, dense dataset exposes catastrophic failures of the unbudgeted baseline. The fidelity audit limits attribution: procedural retrieval is defective, several mechanisms are unexercised, and graph utility is not established. In a complementary matched public diagnostic, ICE v2 loses decisively to pure vector-RAG on LongMemEval: 50.8% versus 72.8% in the evidence-only oracle and 43.0% versus 69.5% in full-S. Paired differences are -22.0 points (95% CI [-26.6, -17.4]) and -26.5 ([-31.3, -21.8]). Conservative abstention accompanies severe multi-session and temporal failures. ICE uses less context in this diagnostic, establishing a quality-cost trade-off rather than superior efficiency. Together, replay, fidelity auditing, and public endpoint testing expose distinct failure modes that neither architectural descriptions nor aggregate scores identify alone.
37 pages. Code and evaluation artifacts: this https URL. The exact system snapshot used for the reported results is preserved in the "v2-paper-eval" tagged release
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR)
arXiv:2609.16730 [cs.AI]
(or arXiv:2609.16730v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.16730 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 271 cuvinte
Subiecte conexe
Hyperloom: A Multi-Agent Harness for Autonomous Inference Optimization on AMD GPUs
AMD · 21 sept. 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · 2 oct. 2026
Agent Harness vs Agent Framework vs MCP: Which Layer Owns the Loop, State, Tools, Permissions, and Recovery
MarkTechPost · 14 sept. 2026
How Intuit built an agentic disaster recovery assistant with Amazon Bedrock
AWS · 4 sept. 2026