Skip to content
aiqbench takeAI Safety

aiqbench analysis

This research identifies a behavioral invariant in LLM agents where successful memory poisoning attacks mechanically require specific tool invocation sequences, enabling high-accuracy detection (AUC > 0.99) across diverse model scales. However, the study reveals a critical deployment boundary where legitimate memory-grounded actions trigger false positives unless gated with recipient metadata, fundamentally limiting standalone blocking strategies. The work establishes that these signatures are attack preconditions rather than direct maliciousness predicates, necessitating nuanced architectural safeguards for agent safety.
Source coveringarXiv
What this isA model's reading of the article and related coverage, written for aiqbench. Not the publisher's words.

Forensic Trajectory Signatures for Agent Memory Poisoning Detection

arXiv3 min read

arXiv:2606.30566v2 Announce Type: replace-cross Abstract: We discover a behavioral invariant in LLM agents under persistent memory poisoning and characterize its deployment boundary. In architectures where retrieval is routed through observable memory-tool invocations, successful attacks require calling memory_recall_fact before email_send_email, a transition mechanistically forced by the attack's information-retrieval dependency. A simple rule exploiting this invariant achieves AUC = 0.9563; a Random Forest over 19 trajectory features refines it to AUC = 0.9904 (BCa 95% CI [0.987, 0.993]). The signature is overdetermined within the poisoned-but-defended evaluation set: removing all recall-related features leaves AUC unchanged. Cross-model hold-out on 9 models (7B-120B) confirms AUC = 1.000 on 6/9 splits, and the invariant transfers to frontier models (GPT-4.1, GPT-4o) without retraining. [v2] A preregistered follow-up (N=4,360, 13 models) reveals a critical deployment boundary: benign memory-grounded sends produce the same recall_before_send signature, yielding 100% false positives conditional on recall_before_send=1 (unconditional benign FPR: 24.7-52.6% depending on recall protocol). The signature is a valid attack precondition, not a maliciousness predicate; standalone blocking is not viable, but gating with recipient metadata restores separation. A prefix-only variant achieves AUC = 0.934, enabling real-time triage.

Abstract:We discover a behavioral invariant in LLM agents under persistent memory poisoning and characterize its deployment boundary. In architectures where retrieval is routed through observable memory-tool invocations, successful attacks require calling memory_recall_fact before email_send_email, a transition mechanistically forced by the attack's information-retrieval dependency. A simple rule exploiting this invariant achieves AUC = 0.9563; a Random Forest over 19 trajectory features refines it to AUC = 0.9904 (BCa 95% CI [0.987, 0.993]). The signature is overdetermined within the poisoned-but-defended evaluation set: removing all recall-related features leaves AUC unchanged. Cross-model hold-out on 9 models (7B-120B) confirms AUC = 1.000 on 6/9 splits, and the invariant transfers to frontier models (GPT-4.1, GPT-4o) without retraining. [v2] A preregistered follow-up (N=4,360, 13 models) reveals a critical deployment boundary: benign memory-grounded sends produce the same recall_before_send signature, yielding 100% false positives conditional on recall_before_send=1 (unconditional benign FPR: 24.7-52.6% depending on recall protocol). The signature is a valid attack precondition, not a maliciousness predicate; standalone blocking is not viable, but gating with recipient metadata restores separation. A prefix-only variant achieves AUC = 0.934, enabling real-time triage.

16 pages, 3 figures. v2: adds preregistered deployment boundary study (N=4,360, 13 models). Companion to arXiv:2605.08442

Cryptography and Security (cs.CR); Machine Learning (cs.LG)

arXiv:2606.30566 [cs.CR]

(or arXiv:2606.30566v2 [cs.CR] for this version)

https://doi.org/10.48550/arXiv.2606.30566 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Full text under the publisher's open licence · 291 words

Related coverage