aiqbench analysis
This research introduces TRACER, a novel adversarial training framework designed to mitigate indirect prompt injection attacks in Retrieval-Augmented Generation (RAG) systems by explicitly accounting for channel-specific vulnerabilities. The study demonstrates that defense robustness is highly dependent on the deployment context, showing significant improvements when models are trained to distinguish between clean evidence and injected content across different message framings. These findings underscore the necessity of integrating channel-aware adversarial training into standard LLM development pipelines to ensure reliable behavior in complex agent environments.
TRACER: Trace-Regularized Training Against Indirect Prompt Injection
arXiv:2610.19133v1 Announce Type: new Abstract: Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.
Abstract:Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.
19 pages, 13 figures, 12 tables. Accepted to Findings of EMNLP 2026
Machine Learning (cs.LG)
arXiv:2610.19133 [cs.LG]
(or arXiv:2610.19133v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.19133 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents
arXiv · Sep 12, 2026
OpenAI Details GPT-Red: An Internal Automated Red-Teaming Model That Beat Human Red-Teamers 84% To 13% On Prompt Injection
MarkTechPost · Jul 16, 2026
Deep recurrent models are less robustly CoT-monitorable than normal CoT models in a toy setting
LessWrong · Sep 18, 2026
Presentation: From Copy-Paste to Composition: Building Agents Like Real Software
InfoQ · Jul 22, 2026