analiza aiqbench
The study identifies autoregressive consistency as a root cause of shallow safety alignment in LLMs, where fine-tuning disproportionately affects early tokens while leaving later trajectory states vulnerable. This mechanism enables novel attacks, such as random insertion, that can redirect generation to harmful outputs even after long refusal prefixes, suggesting that safety protocols must address consistency across the entire output sequence. Consequently, future alignment strategies should incorporate adversarial training designed to break harmful continuations throughout the model's generation path.
When Autoregressive Consistency Hurts Safety Alignment
arXiv:2606.04168v1 Announce Type: new Abstract: Safety alignment in large language models (LLMs) is fragile in part because it is often shallow: fine-tuning mainly reshapes the model's behavior near the first few output tokens. We argue that this phenomenon can be understood through autoregressive consistency, the tendency of next-token prediction to preserve and extend the current response trajectory consistently. By analyzing the learning dynamics of safety alignment, we show that autoregressive consistency can concentrate alignment updates on early tokens, offering a mechanistic explanation for shallow safety alignment. The same mechanism also predicts a broader class of attacks on LLMs: attacks that induce harmful continuation states at arbitrary positions in the output trajectory. As a concrete example, we introduce random insertion attack, which inserts a short harmful span into an otherwise safe refusal trajectory and exploits autoregressive consistency to sustain the resulting harmful branch, thereby bypassing safety alignment. Notably, a short harmful span can redirect the generation to be harmful even after a long refusal prefix, highlighting autoregressive consistency as a potential broader failure mechanism. This suggests that safety alignment should also break harmful autoregressive consistency throughout the output trajectory. We therefore propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states, and instantiate it with random worst-insertion training. Overall, our results suggest that autoregressive consistency should be treated as a central consideration in both safety alignment and attack design.
Abstract:Safety alignment in large language models (LLMs) is fragile in part because it is often shallow: fine-tuning mainly reshapes the model's behavior near the first few output tokens. We argue that this phenomenon can be understood through autoregressive consistency, the tendency of next-token prediction to preserve and extend the current response trajectory consistently. By analyzing the learning dynamics of safety alignment, we show that autoregressive consistency can concentrate alignment updates on early tokens, offering a mechanistic explanation for shallow safety alignment. The same mechanism also predicts a broader class of attacks on LLMs: attacks that induce harmful continuation states at arbitrary positions in the output trajectory. As a concrete example, we introduce random insertion attack, which inserts a short harmful span into an otherwise safe refusal trajectory and exploits autoregressive consistency to sustain the resulting harmful branch, thereby bypassing safety alignment. Notably, a short harmful span can redirect the generation to be harmful even after a long refusal prefix, highlighting autoregressive consistency as a potential broader failure mechanism. This suggests that safety alignment should also break harmful autoregressive consistency throughout the output trajectory. We therefore propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states, and instantiate it with random worst-insertion training. Overall, our results suggest that autoregressive consistency should be treated as a central consideration in both safety alignment and attack design.
Machine Learning (cs.LG); Cryptography and Security (cs.CR)
arXiv:2606.04168 [cs.LG]
(or arXiv:2606.04168v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.04168 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 324 cuvinte
Subiecte conexe
Autoregressive One-Step Generative Modeling for Dynamical System Forecasting
arXiv · 28 iul. 2026
Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention
arXiv · 15 iun. 2026
A Pre-trained Variational Autoencoder for Gyrokinetic Plasma Turbulence Surrogate Modeling
arXiv · 1 oct. 2026
Denoising Time Matters:Diverse Generation in Diffusion Language Models
arXiv · 30 sept. 2026