aiqbench analysis
This paper introduces the Attack Success Probability (ASP) metric to better evaluate prompt injection vulnerabilities in open-source LLMs, addressing the limitations of binary success rates by accounting for model inconsistency. The study identifies a novel "hypnotism" attack capable of inducing objectionable behaviors in aligned models like StableLM2 and Mistral with approximately 90% probability, highlighting that moderately known models are often more susceptible than industry leaders. These findings underscore the critical need for robust mitigation strategies as open-source LLMs scale into high-stakes sectors like finance and healthcare.
From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
arXiv:2505.14368v3 Announce Type: replace-cross Abstract: Recent studies demonstrate that Large Language Models (LLMs) are vulnerable to attacks that generate harmful or sensitive outputs. As open-source LLMs are increasingly adopted in high-impact applications such as finance, law, and healthcare, systematically investigating their security risks is becoming increasingly important towards a trustworthy LLM era. This paper comprehensively studies effective prompt injection attacks against 14 widely used open-source and three closed-source LLMs on five attack benchmarks. Moreover, existing evaluation metrics mostly only consider the attack success rate, overlooking uncertainty in model responses. Our proposed Attack Success Probability (ASP) additionally captures uncertain behaviors for evaluation, where the model may initially refuse a harmful request but subsequently provide harmful guidance or vice versa, reflecting inconsistency and ambiguity in attack feasibility. By systematically analyzing the effectiveness of prompt injection attacks, we propose a straightforward and effective hypnotism attack; results show that this attack causes aligned language models, including StableLM2, Mistral, Openchat, and Vicuna, to generate objectionable behaviors, achieving around 90% ASP. We also find that moderately well-known LLMs exhibit higher vulnerability to prompt injection attacks, highlighting the need to raise public awareness and prioritize efficient mitigation strategies.
Abstract:Recent studies demonstrate that Large Language Models (LLMs) are vulnerable to attacks that generate harmful or sensitive outputs. As open-source LLMs are increasingly adopted in high-impact applications such as finance, law, and healthcare, systematically investigating their security risks is becoming increasingly important towards a trustworthy LLM era. This paper comprehensively studies effective prompt injection attacks against 14 widely used open-source and three closed-source LLMs on five attack benchmarks. Moreover, existing evaluation metrics mostly only consider the attack success rate, overlooking uncertainty in model responses. Our proposed Attack Success Probability (ASP) additionally captures uncertain behaviors for evaluation, where the model may initially refuse a harmful request but subsequently provide harmful guidance or vice versa, reflecting inconsistency and ambiguity in attack feasibility. By systematically analyzing the effectiveness of prompt injection attacks, we propose a straightforward and effective hypnotism attack; results show that this attack causes aligned language models, including StableLM2, Mistral, Openchat, and Vicuna, to generate objectionable behaviors, achieving around 90% ASP. We also find that moderately well-known LLMs exhibit higher vulnerability to prompt injection attacks, highlighting the need to raise public awareness and prioritize efficient mitigation strategies.
Cryptography and Security (cs.CR); Computation and Language (cs.CL)
arXiv:2505.14368 [cs.CR]
(or arXiv:2505.14368v3 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2505.14368 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words
Related coverage
Frontier Risk Report (February to March 2026)
METR · May 19, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026
From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
arXiv · Jun 17, 2026