aiqbench analysis
This study reveals critical gaps in current LLM safety protocols by demonstrating that models exhibit varying robustness when faced with emoji-augmented adversarial prompts compared to standard text inputs. The findings suggest that relying solely on text-based evaluations significantly underrepresents potential vulnerabilities, particularly across different open-source architectures like Mistral and Gemma versus Qwen. Consequently, the research community must expand its evaluation frameworks to include multimodal input representations to ensure comprehensive security assessments.
Are LLMs Safe Beyond Text: Do Emojis Expose Gaps in Safety Evaluation
arXiv:2608.18164v1 Announce Type: cross Abstract: Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test ($\chi^2 = 32.94, p < 0.001$) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.
Abstract:Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test ($\chi^2 = 32.94, p < 0.001$) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.
3 pages. Accepted at ACL 2026 Workshop on Evaluation in Practice: Methodological Rigor, Sociotechnical Perspectives, & Community Collaboration (EvalEval)
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)
arXiv:2608.18164 [cs.CL]
(or arXiv:2608.18164v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.18164 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 164 words
Related coverage
If This Were a Test, How Much Would It Cost?
LessWrong · Jun 16, 2026
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
The basic argument for making AI better at moral philosophy
LessWrong · Oct 14, 2026