aiqbench analysis
This research proposes a novel retrieval mechanism for memory-augmented assistants that leverages sparse token-level matching rather than relying solely on single-vector embeddings. By utilizing contextual representations already computed by existing models, the method improves recall metrics across multiple benchmarks without requiring additional training of the core encoder. The approach demonstrates significant gains in question-answering accuracy on long-term memory tasks, offering a more efficient path to better evidence retrieval in AI agents.
Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence
arXiv:2610.13277v1 Announce Type: cross Abstract: Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.
Abstract:Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.
Information Retrieval (cs.IR); Computation and Language (cs.CL)
arXiv:2610.13277 [cs.IR]
(or arXiv:2610.13277v1 [cs.IR] for this version)
https://doi.org/10.48550/arXiv.2610.13277 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 256 words
Related coverage
Presentation: From Models to Agents: Building Context-Aware Consumer AI at Scale at DoorDash
InfoQ · Aug 15, 2026
Scaling Agentic AI Factories Through Extreme Co-Design with NVIDIA BlueField
NVIDIA · Jul 16, 2026
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · Aug 18, 2026
Intelligence is Free, Now What? <br> Data Systems for, of, and by Agents
BAIR Blog · Jul 7, 2026