AI Security Engineer

Security starts with a list of things worth stealing

Every security discipline begins the same way: name the assets, name the people who want them, and only then talk about attacks. AI systems are no different, but the asset list is longer than most teams expect, and some of the assets are new.

The assets an AI system exposes

• The model weights. Months of compute and data, often the company's core intellectual property. Stealable by a file copy, or more slowly by querying the API and training a copy (module 4).
• The training data. Frequently more sensitive than the weights — customer records, medical notes, proprietary documents. A model can memorise and regurgitate parts of it.
• The system prompt and tool definitions. Not secret in the cryptographic sense, but they reveal business logic and the shape of the attack surface.
• The tool credentials. An agent that can read email or call an internal API holds a token for it. Whoever controls the agent's behaviour controls that token's use.
• The retrieval corpus. In a RAG system, whatever lands in the index becomes "trusted context" for the model — including things an attacker arranged to put there.
• The outputs. Decisions, code, summaries, actions. Corrupting an output is often the attacker's real goal; the model is just the route.

Who wants them

Attackers differ in what they can touch. A user of your chatbot can only type into it and read what comes back — but that is enough for direct prompt injection and for extraction over time. A content author anywhere on the web can plant text that your system will one day retrieve: an indirect attacker who never logs in. A supply-chain insider — the maintainer of a dataset, a model hub account, a pip package — can reach the training pipeline. A compromised tool or vendor can return poisoned results to an agent. Write these down for your own system; the attacker you forget is the one who succeeds.

Trust boundaries are where the work is

Draw the data-flow: user, application, model, retrieval store, tools, external services. Every arrow that crosses from a less-trusted zone to a more-trusted one is a trust boundary, and each boundary deserves one question: what does the receiving side assume about this input, and is that assumption enforced? The characteristic AI failure is that the model receives text from many zones — the system prompt from the developer, the question from the user, the retrieved chunks from the web — and treats all of it as one undifferentiated prompt. The boundaries exist in your diagram but not in the model.

Two maps to navigate by

OWASP Top 10 for LLM Applications is the practitioner's checklist: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Its value is as a shared vocabulary for a design review — "this is LLM01 and LLM06 together" says a lot in few words.

MITRE ATLAS is the adversary's map: a matrix of tactics (reconnaissance, resource development, initial access, ML attack staging, exfiltration, impact) with techniques under each, built from documented real-world incidents. Where OWASP tells you what categories of weakness exist, ATLAS tells you how an operation is sequenced. Use OWASP to review a design; use ATLAS to write a detection or an incident timeline.

NIST AI 100-2 gives the precise terminology — evasion, poisoning, privacy, misuse — and distinguishes predictive from generative AI attacks. When two engineers argue about whether something is "poisoning" or "injection", this is the tie-breaker.

Worked example: a support chatbot with document search

A retail company ships a chatbot that answers questions from its help centre and, for logged-in users, from their order history.

1. Assets: order data (customer PII), the help-centre index, the order-lookup API token, the system prompt.
2. Attackers: any visitor (direct injection, extraction), a malicious help-centre contributor or a compromised CMS (indirect injection through the index), a scraper (unbounded consumption).
3. Trust boundaries: user input to model; retrieved chunk to model; model output to the order API; model output to the user's browser.
4. OWASP tags: LLM01 (prompt injection) at both input boundaries; LLM02 (sensitive disclosure) at the output; LLM06 (excessive agency) if the order API allows writes; LLM05 (improper output handling) if the answer is rendered as HTML.
5. ATLAS: reconnaissance via the public chatbot, initial access via a crafted help article, exfiltration via a link the model is induced to render.

Notice the diagram found the two most dangerous facts before any testing: the order API allows writes, and the output is rendered as HTML. Both are architectural, and both are cheap to fix at design time.

What is still unsettled

The frameworks are young and they overlap. OWASP's list is revised roughly yearly; ATLAS adds techniques as incidents are reported. Neither tells you how likely an attack is for your system — that judgement is still yours, and it should rest on who can reach each boundary.

What to remember

• List assets and attackers before attacks; the model is rarely the asset the attacker wants.
• Every arrow in the data-flow that crosses trust levels is a boundary the model will not enforce for you.
• OWASP LLM Top 10 for design review; MITRE ATLAS for detection and incident narratives; NIST AI 100-2 for vocabulary.
• The most valuable findings usually appear in the diagram, not in testing.