analiza aiqbench
RUMBA introduces a novel benchmark addressing the critical gap in evaluating long-term conversational memory within Large Language Models, specifically targeting non-English contexts like Russian while offering an aligned English subset. By incorporating fine-grained taxonomies for temporal reasoning and session scope, this tool enables researchers to diagnose specific failure modes in current models that rely on aggregate retrieval metrics rather than deep contextual interaction. This advancement is essential for developing robust AI systems capable of maintaining coherent knowledge across extended user sessions beyond simple recall tasks.
RUMBA: Russian User Memory Benchmark
arXiv:2607.21447v1 Announce Type: new Abstract: The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.
Abstract:The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2607.21447 [cs.CL]
(or arXiv:2607.21447v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.21447 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 242 cuvinte
Subiecte conexe
Links #3: 2026/06 Part 1
LessWrong · 15 iun. 2026
Weekly Cyber Security Newsletter Bulletin – Certighost Exploit, Checkpoint 0-day, HTTP/2 Flaw, Notepad++ Plugin Abuse +20 Stories
Cybersecurity News · 26 iul. 2026
Introducing the Third Generation of Apple’s Foundation Models
Apple · 8 iun. 2026
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
arXiv · 26 aug. 2026