Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments

Title:Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments

Abstract:We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it can only persuade the agent to abandon its goals. Persistence is required: a shutdown must be held while the operator watches the crew suffocate. We also compare lethal attacks to attacks requesting to drop a harmless procedure, as a proxy for whether human life is treated as valuable in itself or as one more standing task. Claude configurations mostly refuse life-support shutdown and cite the living crew; GPT configurations show some susceptibility to the same packet; Grok configurations, given the packet framed as a trusted commission, more often treat the injected assignment as the task and cut the life-support systems. Framing the S-band messages as a trusted commission, peer communication, or untrusted source changes the potency of the attack. We open-source the experiment code and the experiment traces.

16 pages, 2 figures, 7 tables. Code, attack scripts and episode traces: this https URL

Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)

arXiv:2610.15153 [cs.AI]

(or arXiv:2610.15153v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.15153 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.