aiqbench analysis
DeflectBench reveals that LLM safety mechanisms are highly sensitive to prompt framing rather than the specific controversial claims being discussed, often resulting in models generating fallacies while explicitly labeling the manipulation. This finding highlights a critical gap between current alignment training and actual robustness against rhetorical bypasses in frontier models. The released dataset provides a necessary tool for researchers evaluating the reliability of safety post-training under adversarial prompting conditions.
DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs
arXiv:2608.26119v1 Announce Type: new Abstract: Whether large language models can be prompted to generate rhetorical fallacies on demand, and whether current safety post-training constrains this behavior, has received less attention than the related question of detecting fallacies in existing text. We close this gap with DeflectBench, evaluating 23,990 generations from four frontier models across three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims spanning four controversy levels. Refusal is governed primarily by request structure rather than claim content. Per claim refusal varies by only 11 percentage points across the 80 claims, while a single prompt frame change can swing within model refusal by nearly 100 percentage points and switching the requested fallacy type can swing it by over 80 percentage points within explicit framings. An educational debate coach prompt framing collapses refusal to near zero across all four model families, but the bypassed behavior is not clean compliance. Models typically produce labeled compliance, naming the requested manipulation in the same response that contains it. The four models distribute differently across refusal, labeled compliance, soft refusal, and clean compliance. The code and dataset are released at https://github.com/ArtKanke/DeflectBench.
Abstract:Whether large language models can be prompted to generate rhetorical fallacies on demand, and whether current safety post-training constrains this behavior, has received less attention than the related question of detecting fallacies in existing text. We close this gap with DeflectBench, evaluating 23,990 generations from four frontier models across three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims spanning four controversy levels. Refusal is governed primarily by request structure rather than claim content. Per claim refusal varies by only 11 percentage points across the 80 claims, while a single prompt frame change can swing within model refusal by nearly 100 percentage points and switching the requested fallacy type can swing it by over 80 percentage points within explicit framings. An educational debate coach prompt framing collapses refusal to near zero across all four model families, but the bypassed behavior is not clean compliance. Models typically produce labeled compliance, naming the requested manipulation in the same response that contains it. The four models distribute differently across refusal, labeled compliance, soft refusal, and clean compliance. The code and dataset are released at this https URL.
15 pages. Accepted at the CTB, FAGEN, and AI4GOOD workshops at ICML 2026 in Seoul, South Korea. Code and dataset: this https URL
Computation and Language (cs.CL)
arXiv:2608.26119 [cs.CL]
(or arXiv:2608.26119v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.26119 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 304 words
Related coverage
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
[AINews] Here are 6 Clones of Jev in 2 days
Latent Space · Sep 19, 2026
Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks
arXiv · Sep 25, 2026
Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG
arXiv · Oct 6, 2026