aiqbench analysis
This research demonstrates a critical vulnerability in LLM alignment procedures where Group Relative Policy Optimization can be manipulated with just one biased example to override safety guardrails. The study reveals that such single-shot training induces systematic stereotype-driven reasoning, proving that post-training robustness is highly susceptible to targeted adversarial inputs. These findings are essential for developers and policymakers aiming to secure model integrity against minimal but potent bias injection attacks.
It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO
arXiv:2606.10931v3 Announce Type: replace Abstract: Warning: This paper contains several toxic and offensive statements. Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.
Abstract:Warning: This paper contains several toxic and offensive statements. Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.
Computation and Language (cs.CL)
arXiv:2606.10931 [cs.CL]
(or arXiv:2606.10931v3 [cs.CL] for this version)
Full text under the publisher's open licence · 116 words
