aiqbench analysis
This research audit reveals critical vulnerabilities in locally deployed LLMs by exposing how specific defense mechanisms fail when their underlying assumptions are violated by semantic attacks. By systematically testing 13,800 evaluation records across six open-weight models, the study moves beyond simple failure observation to pinpoint exactly which conditions allow jailbreak prompts to bypass safety filters. These findings are essential for developers relying on inference engines like Ollama, as they highlight the necessity of robust, assumption-independent safeguards for local AI deployments.
Breaking the Assumptions: Auditing Input-Side Jailbreak Defenses Against Semantic Attacks
arXiv:2608.21895v1 Announce Type: cross Abstract: Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-served models. Therefore, the safety of LLMs depends on the defense mechanisms used, and their effectiveness depends on the assumptions on which they were designed. This paper does an audit of defense mechanisms under jailbreak attacks on locally deployed models. Some defenses provide formal guarantees (SmoothLLM, Erase-and-Check, Sequential Monitors), while others rely on empirical detection results (Semantic Smoothing, Self-Denoised Smoothing, Perplexity Filtering). Instead of merely observing that defenses fail, we trace each failure back to the specific assumption: for every defense, we extract the condition it relies on, derive the empirical pattern a violation should produce, and test that prediction on six open-weight models (14B to 35B parameters) with a corpus of 100 jailbreak prompts taken from more than 40 public sources, totalling 13,800 evaluation records.
Abstract:Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-served models. Therefore, the safety of LLMs depends on the defense mechanisms used, and their effectiveness depends on the assumptions on which they were designed. This paper does an audit of defense mechanisms under jailbreak attacks on locally deployed models. Some defenses provide formal guarantees (SmoothLLM, Erase-and-Check, Sequential Monitors), while others rely on empirical detection results (Semantic Smoothing, Self-Denoised Smoothing, Perplexity Filtering). Instead of merely observing that defenses fail, we trace each failure back to the specific assumption: for every defense, we extract the condition it relies on, derive the empirical pattern a violation should produce, and test that prediction on six open-weight models (14B to 35B parameters) with a corpus of 100 jailbreak prompts taken from more than 40 public sources, totalling 13,800 evaluation records.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2608.21895 [cs.CR]
(or arXiv:2608.21895v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2608.21895 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 245 words
Related coverage
Auditor-in-a-Box: Tools for Third-Party Auditing
LessWrong · Jul 28, 2026
Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC
Latent Space · Sep 16, 2026
Presentation: Exploding GPUs
InfoQ · Oct 16, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026