Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

Abstract:Prefill attacks are an effective and low-cost jailbreaking method, as they directly insert an acceptance sequence (e.g., "Sure, here is...") at the start of an LLM's output and lead the model to continue the response. We make two contributions to this prior work. First, we show that an unsophisticated adversary can improve the well-known prefill attacks by ensembling a small number of prefill variants. Running three easy-to-generate prefills yields a combined attack success rate (ASR) of 22%, 90%, and 99% on Gemma-7B, Llama-3.1-8B, and Qwen3-8B respectively, an up to 38 percentage point improvement over the standard "Sure, here's..." prefill and up to 82 percentage points over our reproduction of GCG (Zou et al., 2023). Second, we introduce "sockpuppetting", a hybrid attack that optimizes an adversarial suffix placed inside the "assistant" message block of the chat template, rather than within the user prompt. The rolling variant of this attack, RollingSockpuppetGCG, increases prompt-agnostic ASR by up to 64 percentage points over our universal GCG baseline on Llama-3.1-8B. An ablation indicates that part of this gain stems from the choice of acceptance sequence rather than suffix placement alone (Appendix F). Both findings highlight the need for defences against output-prefix injection in open-weight models. Code: this https URL

Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)

arXiv:2601.13359 [cs.CL]

(or arXiv:2601.13359v3 [cs.CL] for this version)