aiqbench analysis
This research exposes a critical vulnerability in Large Language Models where over-reliance on security-aligned training data creates new attack vectors via deceptive few-shot learning. The study demonstrates that even advanced models like Deepseek-R1 are highly susceptible to generating harmful outputs when their preference extraction mechanisms are compromised by toxic samples. These findings underscore the necessity of rigorous auditing for security-related data sources during the RLHF and training phases to prevent model overtrust.
FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-shots Learning
arXiv:2410.08776v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) face significant security risks despite their advanced capabilities. While techniques like Reinforcement Learning with Human Feedback (RLHF) improve ethical alignment, excessive exposure to security-related training data may cause LLMs to overtrust such information, creating new vulnerabilities. Investigating this issue, we propose a novel attack method termed FATS (Feign Agent Attack with Toxic-shots). By obfuscating preference extraction, compromising toxicity samples, and inducing malicious behavior, we can effectively mislead LLMs into generating harmful outputs. To evaluate FATS effectiveness, we introduce the FAQuery dataset and conduct experiments on various LLMs. Well-known benchmarks like Advbench were selected to assess the approach. Results demonstrate that mainstream models, including GPT-4.1 (61.6\%) and Deepseek-R1 (99.3\%) are highly susceptible. It underscored the need to rigorously analyze security-related data sources during model training, developing more secure and reliable LLMs.
Abstract:Large Language Models (LLMs) face significant security risks despite their advanced capabilities. While techniques like Reinforcement Learning with Human Feedback (RLHF) improve ethical alignment, excessive exposure to security-related training data may cause LLMs to overtrust such information, creating new vulnerabilities. Investigating this issue, we propose a novel attack method termed FATS (Feign Agent Attack with Toxic-shots). By obfuscating preference extraction, compromising toxicity samples, and inducing malicious behavior, we can effectively mislead LLMs into generating harmful outputs. To evaluate FATS effectiveness, we introduce the FAQuery dataset and conduct experiments on various LLMs. Well-known benchmarks like Advbench were selected to assess the approach. Results demonstrate that mainstream models, including GPT-4.1 (61.6\%) and Deepseek-R1 (99.3\%) are highly susceptible. It underscored the need to rigorously analyze security-related data sources during model training, developing more secure and reliable LLMs.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
arXiv:2410.08776 [cs.CR]
(or arXiv:2410.08776v3 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2410.08776 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.1007/978-981-95-7078-2_47 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 240 words
Related coverage
Friendly Fire: Hijacking Defensive Cyber AI Agents for Remote Code Execution
AI Now Institute · Jul 8, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026
Frontier Risk Report (February to March 2026)
METR · May 19, 2026