aiqbench analysis
The introduction of the TIER benchmark addresses a critical gap in LLM safety evaluation by moving beyond binary metrics to analyze how models respond to prompts with varying degrees of threat implicitness. Research findings indicate that safety behaviors evolve gradually across different threat levels rather than switching abruptly, revealing significant robustness gaps specifically when facing sophisticated jailbreak attempts. This behavior-aware approach provides essential insights for refining alignment strategies and understanding the nuanced dynamics of model compliance under attack.
TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors
arXiv:2609.05117v1 Announce Type: cross Abstract: Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts yield the most diverse behaviors, while jailbreaks reveal the largest robustness gaps. Furthermore, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.
Abstract:Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We introduce TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of LLMs. TIER covers four risk domains and four threat levels, from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts yield the most diverse behaviors, while jailbreaks reveal the largest robustness gaps. Furthermore, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2609.05117 [cs.CR]
(or arXiv:2609.05117v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2609.05117 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 216 words
Related coverage
[AINews] OpenAI shuts off Cursor
Latent Space · Aug 29, 2026
Current AIs out-persuade professionals in lab settings but (probably) not in the world
LessWrong · Oct 5, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
News
Quantum Computing Report · Jun 7, 2026