aiqbench analysis
This study establishes a rigorous framework for evaluating LLM prompt security by systematizing 272 existing studies into unified taxonomies of attacks, defenses, and vulnerabilities. By releasing open-source datasets and an evaluation protocol that accounts for model-specific dependencies and judger behavior, the work challenges context-free method rankings and highlights how defensive measures can inadvertently degrade benign utility. These findings are critical for researchers aiming to build robust safety benchmarks that reflect real-world deployment complexities rather than isolated attack success rates.
SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses
arXiv:2510.15476v4 Announce Type: replace-cross Abstract: Prompt-security claims for large language models (LLMs) are often compared as properties of individual attacks or defenses, although measured harmfulness depends on the target model, access setting, attack and defense configuration, prompt sample, and judger. We systematize 272 studies through three linked, multi-label taxonomies of jailbreak attacks, defenses, and model vulnerabilities. To make these dependencies explicit, we introduce an access-compatible evaluation protocol over configured $\langle M,A,D,S,J\rangle$ tuples and instantiate it in PromptSecurity, an open-source platform that records method settings, outputs, judgments, and query costs. We release two complementary datasets: JailbreakDB, with 445,752 jailbreak system-user prompt pairs and 1,094,122 benign prompts, and PromptSecurity-Eval, with 166,200 harmful-query and 9,200 benign-utility output records. We evaluate 11 target models, 20 attacks, nine defenses, and three automated judgers under a matched, single-turn protocol using 100 harmful queries and separate benign-utility tasks. Lower attack success can reflect failure to execute transformed requests rather than stronger safety; attack-category advantages can reverse when one method is excluded, and attack success varies with assistant choice. Defenses can reduce harmful outputs while degrading benign-task utility or causing backfire. Rejudging identical outputs also shows that measured attack success is sensitive to the behavior supplied to the judger, highlighting the need to report judger configurations alongside ASR. These findings motivate configuration-aware reporting rather than context-free method rankings. Leaderboard: https://datasec-lab.github.io/PromptSecurityLeaderboard/. Dataset: https://huggingface.co/datasets/youbin2014/JailbreakDB. GitHub: https://github.com/datasec-lab/PromptSecurity.
Abstract:Prompt-security claims for large language models (LLMs) are often compared as properties of individual attacks or defenses, although measured harmfulness depends on the target model, access setting, attack and defense configuration, prompt sample, and judger. We systematize 272 studies through three linked, multi-label taxonomies of jailbreak attacks, defenses, and model vulnerabilities. To make these dependencies explicit, we introduce an access-compatible evaluation protocol over configured $\langle M,A,D,S,J\rangle$ tuples and instantiate it in PromptSecurity, an open-source platform that records method settings, outputs, judgments, and query costs. We release two complementary datasets: JailbreakDB, with 445,752 jailbreak system-user prompt pairs and 1,094,122 benign prompts, and PromptSecurity-Eval, with 166,200 harmful-query and 9,200 benign-utility output records. We evaluate 11 target models, 20 attacks, nine defenses, and three automated judgers under a matched, single-turn protocol using 100 harmful queries and separate benign-utility tasks. Lower attack success can reflect failure to execute transformed requests rather than stronger safety; attack-category advantages can reverse when one method is excluded, and attack success varies with assistant choice. Defenses can reduce harmful outputs while degrading benign-task utility or causing backfire. Rejudging identical outputs also shows that measured attack success is sensitive to the behavior supplied to the judger, highlighting the need to report judger configurations alongside ASR. These findings motivate configuration-aware reporting rather than context-free method rankings. Leaderboard: this https URL. Dataset: this https URL. GitHub: this https URL.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2510.15476 [cs.CR]
(or arXiv:2510.15476v4 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2510.15476 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 323 words
Related coverage
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · Oct 5, 2026
Article: Governing AI in the Cloud: A Practical Guide for Architects
InfoQ · Jun 15, 2026
SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
arXiv · Sep 14, 2026
Update on Security at METR
METR · Aug 31, 2026