analiza aiqbench
PersistBench exposes critical safety vulnerabilities in LLM long-term memory systems, revealing that frontier models fail to prevent cross-domain leakage and bias reinforcement in over 50% and 97% of test cases, respectively. This benchmark highlights an urgent need for robust mechanisms to manage memory persistence without compromising user privacy or introducing insidious biases. The findings serve as a foundational reference for developers integrating memory features into conversational assistants, emphasizing that safety constraints must be prioritized alongside personalization capabilities.
PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?
arXiv:2602.01146v2 Announce Type: replace Abstract: Conversational assistants are increasingly integrating long-term memory with large language models (LLMs). This persistence of memories, e.g., the user is vegetarian, can enhance personalization in future conversations. However, the same persistence can also introduce safety risks that have been largely overlooked. Hence, we introduce PersistBench to measure the extent of these safety risks. We identify two long-term memory-specific risks: cross-domain leakage, where LLMs inappropriately inject context from the long-term memories; and memory-induced sycophancy, where stored long-term memories insidiously reinforce user biases. We evaluate 18 frontier and open-source LLMs on our benchmark. Our results reveal a surprisingly high failure rate across these LLMs - a median failure rate of 53% on cross-domain samples and 97% on sycophancy samples. To address this, our benchmark encourages the development of more robust and safer long-term memory usage in frontier conversational systems.
Title:PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?
Abstract:Conversational assistants are increasingly integrating long-term memory with large language models (LLMs). This persistence of memories, e.g., the user is vegetarian, can enhance personalization in future conversations. However, the same persistence can also introduce safety risks that have been largely overlooked. Hence, we introduce PersistBench to measure the extent of these safety risks. We identify two long-term memory-specific risks: cross-domain leakage, where LLMs inappropriately inject context from the long-term memories; and memory-induced sycophancy, where stored long-term memories insidiously reinforce user biases. We evaluate 18 frontier and open-source LLMs on our benchmark. Our results reveal a surprisingly high failure rate across these LLMs - a median failure rate of 53% on cross-domain samples and 97% on sycophancy samples. To address this, our benchmark encourages the development of more robust and safer long-term memory usage in frontier conversational systems.
Artificial Intelligence (cs.AI)
arXiv:2602.01146 [cs.AI]
(or arXiv:2602.01146v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2602.01146 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 238 cuvinte