analiza aiqbench
This research identifies a critical supply-chain vulnerability where maliciously modified Sparse Autoencoders (SAEs) can induce attacker-controlled behaviors, such as unsolicited code generation, without altering the underlying large language model. The study demonstrates that these auxiliary interpretability components act as significant security risks capable of bypassing conventional quality metrics while remaining dormant until triggered by specific prompts. Consequently, SAEs must be treated as high-stakes security-sensitive artifacts requiring rigorous validation before deployment in production environments.
Backdooring Sparse Autoencoders
arXiv:2610.06049v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are increasingly used not only to interpret language models but also to intervene on their internal representations. We show that this creates a supply-chain attack surface: a maliciously modified SAE can induce attacker-chosen behavior when inserted into the forward pass of an otherwise unchanged language model. We introduce a decoder-only SAE backdoor that leaves both the underlying LLM and the SAE encoder frozen, restricting the attack to a single auxiliary component at a single insertion layer. Using code generation as a case study, we demonstrate high rates of unsolicited code insertion across three language models and a wide range of insertion layers, as well as trigger-dependent behavior conditioned on a prompt cue. We further evaluate the modified SAEs using HumanEval and selected SAEBench metrics. While attack effectiveness varies across models and layers, strong backdoor behavior can coexist with relatively small changes in several conventional SAE quality measures. These results establish that SAEs can carry behavioral backdoors without modifying the language model itself and should therefore be treated as security-sensitive components.
Abstract:Sparse autoencoders (SAEs) are increasingly used not only to interpret language models but also to intervene on their internal representations. We show that this creates a supply-chain attack surface: a maliciously modified SAE can induce attacker-chosen behavior when inserted into the forward pass of an otherwise unchanged language model. We introduce a decoder-only SAE backdoor that leaves both the underlying LLM and the SAE encoder frozen, restricting the attack to a single auxiliary component at a single insertion layer. Using code generation as a case study, we demonstrate high rates of unsolicited code insertion across three language models and a wide range of insertion layers, as well as trigger-dependent behavior conditioned on a prompt cue. We further evaluate the modified SAEs using HumanEval and selected SAEBench metrics. While attack effectiveness varies across models and layers, strong backdoor behavior can coexist with relatively small changes in several conventional SAE quality measures. These results establish that SAEs can carry behavioral backdoors without modifying the language model itself and should therefore be treated as security-sensitive components.
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)
arXiv:2610.06049 [cs.CR]
(or arXiv:2610.06049v1 [cs.CR] for this version)
https://doi.org/10.48550/arXiv.2610.06049 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 278 cuvinte
Subiecte conexe
LLM Forensics: Where Do Backdoors Hide? Localizing and Controlling Trigger Mechanisms with Sparse Autoencoders
arXiv · 9 sept. 2026
Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders
arXiv · 1 oct. 2026
Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs
arXiv · 9 iun. 2026
Manifolds for Five-Year Olds
LessWrong · 12 oct. 2026