An AI Agent Execution Environment to Safeguard User Data

Abstract:AI agents promise to serve as general-purpose personal assistants for their users, which requires them to have access to private user data (e.g., personal and financial information). This poses a serious risk to security and privacy: an AI model may hallucinate or make mistakes, and adversaries may attack it (e.g., via prompt injection) to exfiltrate user data. This paper presents GAAP (Guaranteed Accounting for Agent Privacy), an execution environment for AI agents that guarantees confidentiality for private user data. Crucially, GAAP provides this guarantee deterministically, without trusting the agent with private user data, and without requiring any AI model or the user prompt to be free of attacks. Through dynamic and directed user prompts, GAAP collects permission specifications from users describing how their private data may be shared. GAAP then enforces that the agent's data disclosures comply with these specifications by tracking how the AI agent accesses and uses private user data. GAAP augments Information Flow Control with novel persistent data stores and annotations that enable tracking the private information flow both across steps of a single task and over multiple separate tasks. Our evaluation confirms that GAAP blocks all data disclosure attacks, including those that make other state-of-the-art systems disclose private user data to untrusted parties, with only a small impact on agent utility.

Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Operating Systems (cs.OS)

arXiv:2604.19657 [cs.CR]

(or arXiv:2604.19657v2 [cs.CR] for this version)

https://doi.org/10.48550/arXiv.2604.19657 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.