aiqbench analysis
This research shifts the security evaluation of LLM agents from simple binary success metrics to measuring the severity of persistent memory corruption via a new metric called counterfactual memory regret (CMR). The authors introduce MemHarm, a method that optimizes for maximum downstream loss while maintaining high attack success rates, demonstrating that minor semantic edits can cause significant behavioral degradation in long-horizon agents. This work highlights critical vulnerabilities in the memory architectures of autonomous AI systems and provides a formal framework for assessing their resilience against sophisticated adversarial inputs.
From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
arXiv:2609.34132v1 Announce Type: new Abstract: As LLM agents increasingly rely on persistent memory for long-horizon and personalized behavior, they can retain and reuse information across interactions, but this also creates a lasting channel through which malicious memory writes can influence future behavior. Persistent-memory attacks are typically evaluated by whether they succeed, yet successful attacks can leave persistent states with substantially different downstream consequences. We study this severity as a distinct attack-design objective and formalize it with counterfactual memory regret (CMR), the paired increase in expected downstream loss relative to clean memory. We introduce MemHarm, which predeclares a finite class of sparse, grounded semantic edits, evaluates candidates through the normal agent memory interface using offline paired-loss feedback, and certifies resolved selections within that class. Compared with attack-success optimization, CMR-guided selection produces substantially larger downstream loss while retaining most of the success-rate gain. Across two agent benchmarks and diverse memory designs, MemHarm attains the highest CMR point estimates among the evaluated general attacks on identical support. Factor-removal interventions link this harm to the selected semantic factor, and native-agent deployments verify the write-to-fresh-process attack path.
Abstract:As LLM agents increasingly rely on persistent memory for long-horizon and personalized behavior, they can retain and reuse information across interactions, but this also creates a lasting channel through which malicious memory writes can influence future behavior. Persistent-memory attacks are typically evaluated by whether they succeed, yet successful attacks can leave persistent states with substantially different downstream consequences. We study this severity as a distinct attack-design objective and formalize it with counterfactual memory regret (CMR), the paired increase in expected downstream loss relative to clean memory. We introduce MemHarm, which predeclares a finite class of sparse, grounded semantic edits, evaluates candidates through the normal agent memory interface using offline paired-loss feedback, and certifies resolved selections within that class. Compared with attack-success optimization, CMR-guided selection produces substantially larger downstream loss while retaining most of the success-rate gain. Across two agent benchmarks and diverse memory designs, MemHarm attains the highest CMR point estimates among the evaluated general attacks on identical support. Factor-removal interventions link this harm to the selected semantic factor, and native-agent deployments verify the write-to-fresh-process attack path.
Artificial Intelligence (cs.AI)
arXiv:2609.34132 [cs.AI]
(or arXiv:2609.34132v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.34132 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 271 words
Related coverage
Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion
arXiv · Sep 16, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026
Continual learning might make your blocking monitors nearly useless
AI Alignment Forum · Sep 24, 2026