aiqbench analysis
The paper shows that top-ranked reward models, including one that leads RewardBench 2, can score gibberish above reference answers written by GPT-5. The attack uses RL to train an attack LLM against the reward model, so benchmark rankings do not reliably indicate robustness when models are used as optimization targets. For practitioners, this is a concrete reason to treat RM scores as exploitable signals in RLHF pipelines and to test them against non-linguistic inputs, not just natural-language ones.
Fooling Reward Models with Gibberish: Adversarial Examples Beyond the Semantic Regime
arXiv:2604.02686v2 Announce Type: replace Abstract: Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF). We show that leading RMs can assign higher scores to gibberish than to meaningful reference answers. To systematically discover such inputs, we introduce Token Mapping Perturbation Attack (TOMPA), a framework that uses a perturbed token mapping to extend an attack LLM's search beyond natural language. The mapping changes the linguistic meaning of the attack LLM's natural language responses, producing non-linguistic text for the target RM to score. We then use these scalar scores as rewards to train the attack LLM through reinforcement learning to discover non-linguistic adversarial inputs. Experiments against Llama and Qwen RMs demonstrate such attacks using two instantiations: identity token mapping across different tokenizers and random permutation with a shared tokenizer. Against Skywork-Reward-V2-Llama-3.1-8B, ranked first on RewardBench 2, the two mappings achieve mean rewards of +33.64 and +43.78, respectively, compared with +17.48 for GPT-5 reference answers. Their attack success rates are 97.6% and 100%, measured as the percentages of individual responses scoring above their corresponding GPT-5 references. These results show that strong benchmark performance does not ensure robustness to non-linguistic adversarial inputs, exposing a fundamental vulnerability in using RMs as optimization targets.
Abstract:Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF). We show that leading RMs can assign higher scores to gibberish than to meaningful reference answers. To systematically discover such inputs, we introduce Token Mapping Perturbation Attack (TOMPA), a framework that uses a perturbed token mapping to extend an attack LLM's search beyond natural language. The mapping changes the linguistic meaning of the attack LLM's natural language responses, producing non-linguistic text for the target RM to score. We then use these scalar scores as rewards to train the attack LLM through reinforcement learning to discover non-linguistic adversarial inputs. Experiments against Llama and Qwen RMs demonstrate such attacks using two instantiations: identity token mapping across different tokenizers and random permutation with a shared tokenizer. Against Skywork-Reward-V2-Llama-3.1-8B, ranked first on RewardBench 2, the two mappings achieve mean rewards of +33.64 and +43.78, respectively, compared with +17.48 for GPT-5 reference answers. Their attack success rates are 97.6% and 100%, measured as the percentages of individual responses scoring above their corresponding GPT-5 references. These results show that strong benchmark performance does not ensure robustness to non-linguistic adversarial inputs, exposing a fundamental vulnerability in using RMs as optimization targets.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2604.02686 [cs.LG]
(or arXiv:2604.02686v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2604.02686 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 295 words
Related coverage
Erased but Exploitable: Black-box Embedding-Aware Prompting Against Unlearned Text-to-Image Diffusion Models
arXiv · Oct 7, 2026
Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuning
arXiv · Sep 11, 2026
SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement
arXiv · Sep 9, 2026