aiqbench analysis
This research introduces LLMail-Inject, a massive dataset of 208,095 human-written indirect prompt injection attacks that exposes critical vulnerabilities in current model defenses. The study demonstrates that models achieving high security scores on static benchmarks suffer significantly higher attack success rates when faced with this diverse, real-world adversarial data, highlighting the limitations of existing training sets for robustness. By releasing a platform with multiple difficulty tiers and rich annotations, the authors provide an essential resource for advancing AI safety against evolving injection vectors.
LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
arXiv:2506.09956v2 Announce Type: replace-cross Abstract: Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100-1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack's complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve 5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign--specifically trained against prompt injection--still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.
Abstract:Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100-1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack's complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve 5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign--specifically trained against prompt injection--still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.
Dataset at: this https URL. Accepted at EMNLP 2026 (Findings)
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2506.09956 [cs.CR]
(or arXiv:2506.09956v2 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2506.09956 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 287 words
Related coverage
Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs
arXiv · Jun 8, 2026
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection
arXiv · Jul 29, 2026
Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs
LessWrong · Jul 27, 2026