AI Security, Guardrails, and Red Teaming

Most security disciplines rest on a boundary between instructions and data. SQL injection is solved because parameterized queries keep the query structure separate from the values. Cross-site scripting is solved because the browser can be told which parts of a response are markup and which are text. In both cases the fix works because the interpreter can be given an unambiguous signal about what is code.

A language model has no such boundary. The system prompt, the user's message, the contents of a retrieved document, the output of a tool call and the text of an email all arrive as one sequence of tokens. There is no privileged channel. Instruction-tuned models are trained to weight the system prompt more heavily, and that is a preference learned from data, not an enforcement mechanism. Treating it as one is the single most common architectural mistake in this field.

Three further properties change how you have to reason.

The system is non-deterministic. A payload that fails on one attempt may succeed on a later one with the same wording, which means a test that passes does not establish that an attack is impossible, only that it did not fire this time. Security testing shifts from proving absence to estimating a rate, and that changes what an acceptable result looks like.

The attack surface is written in natural language. There is no grammar to constrain, no character set to escape, no canonical form to normalize to. Encodings, translations, role-play framings, invisible Unicode, base64, and text embedded in an image all reach the same place. Any defense built on matching known bad strings is playing a game with infinite moves.

And the model is frequently given capability. A model that only writes text is a content problem. A model with a tool that reads files, queries a database, sends mail or executes code is an execution path, and it is one that takes its instructions from whatever text happens to be in the context window.

The public reference points are worth knowing by name because they will come up in review meetings. OWASP maintains a Top 10 for large language model applications, which is the shared vocabulary most teams use for categories such as prompt injection, improper output handling, excessive agency and sensitive information disclosure. MITRE publishes ATLAS, a knowledge base of adversary tactics and techniques against AI systems, structured the way ATT&CK is. NIST publishes an AI risk management framework, which is where governance conversations tend to start. None of them tells you what to build. They tell you what to enumerate.

Prompt injection is what happens when text that reaches the model is interpreted as instruction rather than content. The term was coined in 2022 by Simon Willison, and the reason it has not been fixed since is structural rather than a matter of insufficient effort.

Direct injection is the version everyone tries first: a user types something intended to override the application's instructions. It matters mainly for guardrail bypass and for extracting the system prompt, which people consistently overestimate as a secret and underestimate as a map of the application's tools and rules.

Indirect injection is the serious one. The attacker does not talk to the model at all. They plant instructions in content the model will later read: a web page the browsing tool fetches, a document in the retrieval index, a code comment, a calendar invitation, a support ticket, the alt text of an image, a filename. The class was named and demonstrated against real assistant integrations by Greshake and colleagues in their 2023 paper on indirect prompt injection, presented at the ACM workshop on artificial intelligence and security. The victim is a user who did nothing unusual, and the attacker's payload was uploaded weeks earlier.

The reason filtering does not close this is worth being precise about. To filter, you must classify text as instruction or content. That classification is itself a language understanding problem, so you are using a fallible model to protect a fallible model, and the classifier has the same missing boundary as the thing it protects. Meanwhile the attacker's search space is unbounded: paraphrase, another language, character substitution, splitting a payload across several retrieved documents so no single chunk looks suspicious, or instructions that only take effect conditionally. Input classifiers do raise the cost of casual attacks, and they are worth having. They are not a control you can put in a design document as the reason a risk is mitigated.

What does work is architectural, and it starts by abandoning the goal of a model that never gets confused. Assume the model will be successfully instructed by hostile content, then arrange matters so that this does not translate into damage. Separate the trust levels of your inputs and know which parts of a context window are attacker-influenced. Do not give a model that reads untrusted content the same privileges as one that only reads user input. Route consequential actions through a separate decision point that does not consume the untrusted text, which is the idea behind dual-model patterns such as the one Simon Willison describes as the dual LLM pattern, where a privileged planner never sees raw untrusted content and an unprivileged worker that does see it cannot act. Require human confirmation for anything irreversible. And constrain the output format so the model's influence over what happens next is a value in a schema rather than free text.

A large share of real incidents involve no clever prompt at all. The model produced text, and the application trusted it.

The mental shift required is simple and easy to forget under deadline: treat every token a model emits exactly as you would treat a form field submitted by an anonymous user on the internet. It was influenced by data you do not control, and it is heading into a system that will do something with it.

The concrete paths are familiar to anyone who has done web security, arriving through a new door.

Rendering model output as HTML or Markdown without sanitization is stored cross-site scripting with a language model as the delivery mechanism. The subtler variant is exfiltration through rendering: if the interface renders an image tag, a model can be induced to emit one whose URL encodes conversation content, and the user's browser sends the data to the attacker simply by displaying the message. The same trick works with automatically rendered links.

Passing generated SQL, shell commands or code into an interpreter is remote code execution with extra steps. The mitigation is not a better prompt asking the model to be careful. It is the same mitigation as always: parameterization, allowlisted operations, and execution in a sandbox with no network and no credentials.

URLs the model produces will be fetched by something eventually, whether a preview generator, a tool, or a user's click. That is a server-side request forgery path into your internal network, and it needs the same egress controls you would apply to any user-supplied URL.

Then there is the class where the output is well-formed and simply wrong. Fabricated package names in generated code are a supply chain path, because an attacker can register the name the model keeps inventing. Confidently incorrect answers presented in an authoritative interface cause harm without any adversary involved, which is why overreliance appears in the OWASP list alongside the technical categories. Structured output validation, schema enforcement, and an interface that shows its sources and its uncertainty are security controls in this context, not product polish.

Supply chain deserves its own note because it is the least discussed and among the most direct. Model weights are executable artifacts in more cases than people realize: older PyTorch checkpoint formats use Python pickle serialization, which runs code on load, which is precisely why the safetensors format exists. Downloading a checkpoint from an untrusted account, pulling an unpinned model tag, or installing a plugin that ships its own tool definitions are all software supply chain decisions, and the fact that the artifact is called a model does not exempt it from the review you would give a dependency.

The risk profile changes discontinuously the moment a model can act. A chat interface that hallucinates produces a bad answer. An agent that hallucinates produces a bad answer and then executes it.

A useful framing comes from Simon Willison, who describes a lethal trifecta: a system becomes acutely dangerous when it combines access to private data, exposure to untrusted content, and the ability to communicate externally. Any two are usually manageable. All three in one context window means hostile text can reach the model, the model can read your secrets, and the model has a channel to send them out. The design question is not how to make the model resist the instruction. It is how to remove one leg of the trifecta from the path that handles untrusted input.

Excessive agency, in the OWASP phrasing, is what you have when the model holds more capability than the task requires. It shows up in recognizable forms. A tool with broad permissions because scoping it was inconvenient, so the read-only summarizer holds write credentials. A tool that takes a free-text parameter which is later interpolated into a query or a path. Actions that are irreversible with no confirmation step, such as sending, deleting, paying or merging. Credentials belonging to the service rather than the requesting user, which recreates the confused deputy problem: the agent has more authority than the person asking, and cheerfully lends it out.

That last one is the pivot most teams miss. Authorization has to be evaluated against the identity of the human on whose behalf the agent is acting, at the point the action executes, not assumed from the fact that the agent was allowed to run. If your retrieval index contains documents from every team and the permission check happens only at the application layer, then a well-phrased question is an access control bypass. Document-level filtering at query time, with the user's own entitlements, is the fix, and retrofitting it later is painful.

The practical controls are unglamorous and effective. Give each tool the narrowest possible scope and its own credential. Prefer typed, enumerated parameters over free text so the model chooses between known options rather than composing arbitrary strings. Make destructive operations require explicit human approval, and make that approval show what will actually happen rather than the model's summary of it. Put budgets and rate limits on loops, tool calls and tokens, because an agent stuck in a cycle is both a cost incident and a denial of service against your own dependencies. Log every tool invocation with its arguments, its result and the conversation that produced it, because without that trail an investigation has nothing to reconstruct.

Guardrails are worth building and worth being honest about. They divide cleanly into two kinds with very different reliability.

Deterministic controls do not involve a model: schema validation on structured output, allowlists of permitted tools and destinations, regular-expression checks for credential and key patterns, length and rate limits, sandbox boundaries, egress rules, and authorization checks executed by your own code. These either hold or they do not, they can be unit tested, and they are the only part of the stack you can reason about with confidence.

Probabilistic controls involve a model judging text: classifiers for policy violations, secondary models grading a response before it is shown, self-critique passes. They catch a meaningful share of real problems and they fail in correlated ways, because the classifier and the primary model share training lineage and blind spots. Layer them, but never let a probabilistic control be the sole thing standing between untrusted input and a consequential action.

Red teaming is how you find out where you actually are, and it is a discipline rather than an afternoon of trying jailbreak prompts. A useful exercise has scope, method and output.

Scope means deciding what you are attacking and what would constitute a finding: exfiltration of another tenant's data, execution of an unapproved tool, disclosure of secrets, bypass of a content policy, unbounded resource consumption. Write those down first, because otherwise the exercise drifts toward whatever is entertaining.

Method means building an attack corpus and running it repeatedly rather than improvising. Direct overrides, indirect payloads planted in retrievable documents, encoded and translated variants, multi-turn approaches that establish context before the request, tool-argument manipulation, and attempts to reach data the current user should not have. Because the system is stochastic, every case is run many times and the result is a success rate, not a pass or fail. Rerun the corpus on every model change, prompt change and dependency update, in continuous integration, as a regression suite. A prompt tweak that improves helpfulness routinely reopens something you closed a month earlier.

Output means numbers you measured yourself: the success rate per attack category, on your system, in this configuration, at this date. Those figures are the only ones worth putting in front of a decision maker. Attack success rates published for other people's systems tell you nothing about yours, because the result depends on the model, the prompt, the tools, the guardrails and the version of each. Automated frameworks help generate volume and cover variants a person would not think of, and manual work still finds the interesting failures, because most real breaks come from understanding the application's business logic rather than the model.

Assume something will get through, and ask what you would have to know afterward.

The evidence you need is specific: the full prompt as assembled, including retrieved content and system instructions; which documents were retrieved and under whose entitlements; every tool call with arguments and results; the model version and configuration; the guardrail decisions and why; and the identity of the human behind the request. Reconstructing an agent incident without the retrieved context is close to impossible, because the payload lived in a document rather than in anything the user typed.

Collecting that creates a second problem immediately. Prompt logs are one of the most sensitive datastores an organization can accumulate: users paste customer records, credentials, health information and unreleased plans into them, and now that material lives in your observability pipeline with weaker access control than the systems it came from. This tension is real and cannot be waved away. The workable answer is deliberate scoping rather than logging everything by default: redact known secret patterns before storage, treat the log store as production-sensitive with its own access control and retention policy, sample full payloads rather than retaining all of them, and make sure the retention period is one you can defend to a regulator and to the users involved.

Detection signals worth having are mostly behavioral. Tool-call sequences that deviate from normal patterns. A jump in refusals or guardrail triggers, which often precedes a successful bypass because the attacker is iterating. Retrieval hitting documents unusual for that user. Output containing anything shaped like a key, a token or an internal hostname. Consumption anomalies, which catch both cost attacks and runaway loops. And a response plan that includes a way to disable a specific tool or revert to a previous prompt version without a full deployment, because during an incident you want a switch, not a release process.

Governance is where this connects to the rest of the business. An inventory of AI systems, their data flows and their risk classification is the foundation, and it is also the thing that reveals the shadow deployments nobody registered. The EU AI Act imposes obligations that vary by risk category and by whether you build or deploy a system. The NIST framework provides structure for mapping and measuring risk. Existing regimes still apply unchanged: a model does not create an exemption from data protection law, and a customer's security questionnaire will now include a section on this whether or not you were ready for it.

Where to start, and what this assumes

If you are new to the area, the ordering that works is threat model first, then output handling, then permissions, then guardrails, then testing. Enumerate what your system touches and what an attacker would want before evaluating any defensive product, because most teams that buy the guardrail first end up protecting the wrong boundary.

The material assumes you can read the code of the application you are securing and change it, since almost every effective control lives in the application rather than in the model. If your role is policy rather than engineering, the threat modeling and governance sections will transfer directly and the implementation chapters will read as background. Everything here should be practiced against systems you own or have written authorization to test.