aiqbench analysis
This research introduces ReSA, a novel adversarial framework that exploits latent safety rewards in aligned Large Language Models to generate harmful outputs with high transferability across diverse systems. By leveraging maximum entropy inverse reinforcement learning, the method efficiently bypasses traditional alignment defenses, highlighting a critical vulnerability in current model alignment strategies. This work is essential for understanding the scalability and robustness of LLM safety mechanisms against sophisticated attack vectors.
Reward Stealing Attack on Large Language Models
arXiv:2610.06670v1 Announce Type: new Abstract: Adversarial attacks on Large Language Models (LLMs) aim to induce harmful content. However, existing methods suffer from high computational costs or strict model-pairing dependencies, limiting their scalability and transferability. We propose Reward Stealing Attack (ReSA), an adversarial attack framework that targets the latent safety reward underlying LLM alignment. ReSA employs maximum entropy inverse reinforcement learning to recover a proxy reward model solely from the aligned model's behavior. The extracted reward is then reversed at inference time to derive an adversarial policy, efficiently implemented via a reward-guided decoding mechanism. Experiments demonstrate that a single recovered reward generalizes across prompts and diverse models to reveal a fundamental alignment vulnerability, enabling ReSA to significantly outperform existing attacks in effectiveness and transferability. The code is available at https://github.com/GarminQ/ReSA.
Abstract:Adversarial attacks on Large Language Models (LLMs) aim to induce harmful content. However, existing methods suffer from high computational costs or strict model-pairing dependencies, limiting their scalability and transferability. We propose Reward Stealing Attack (ReSA), an adversarial attack framework that targets the latent safety reward underlying LLM alignment. ReSA employs maximum entropy inverse reinforcement learning to recover a proxy reward model solely from the aligned model's behavior. The extracted reward is then reversed at inference time to derive an adversarial policy, efficiently implemented via a reward-guided decoding mechanism. Experiments demonstrate that a single recovered reward generalizes across prompts and diverse models to reveal a fundamental alignment vulnerability, enabling ReSA to significantly outperform existing attacks in effectiveness and transferability. The code is available at this https URL.
Computation and Language (cs.CL)
arXiv:2610.06670 [cs.CL]
(or arXiv:2610.06670v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.06670 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 221 words
Related coverage
Links #5: 2026/07
LessWrong · Jul 31, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
What We're Up Against: An AI Safety Crash Course
LessWrong · Sep 24, 2026
An unexamined cause of the OpenAI Hugging Face hacking incident: its binary performance metric
LessWrong · Sep 23, 2026