LLM guardrails architecture: input checks, output checks and policy layers

Key points

• A system prompt is an instruction, not a control. Guardrails are code outside the model that can block, change or route what goes in and out.
• Check every input that enters the context, including retrieved documents and tool results, not just the user's message.
• For agents, the tool-call policy matters most: allow, deny or ask a person before each call, with argument checks, rate limits and an audit log.
• Rules are fast and narrow, classifiers are fast and fuzzy, LLM judges are slow and flexible. Layer them, cheapest first.
• Streaming needs a holdback window: check the text accumulated so far and release only what has passed.
• Decide per check whether it fails open or closed, log every verdict, and turn every incident into a red-team test.

Where guardrails sit: before, around and after the model

A guardrail is a check that runs outside the model and can stop, change or reroute what goes in or comes out. Outside is the important word. A system prompt that tells the model to behave is an instruction, and prompt injection exists because instructions can be overridden by other text. OWASP's entry on prompt injection says it is unclear whether fool-proof prevention exists at all, so the design goal is layers that each catch some failures and limit the damage of the rest. A guardrail stack has three places to put checks:

• Before the model: checks on the user's message and on everything else that enters the context, such as retrieved documents, tool results, uploaded files and earlier agent output. Greshake and colleagues showed that indirect prompt injection arrives through data the application retrieves, with no need for the attacker to talk to the model directly.
• Around tool calls: a policy layer between what the model asks for and what actually runs. For an agent this is where most of the risk sits.
• After the model: checks on the output before it reaches a user, a browser, a database or another agent.

NVIDIA's open-source NeMo Guardrails toolkit names five kinds of rails along the same path: input rails that validate user input before the model is called, retrieval rails that filter retrieved chunks, dialog rails that steer a multi-turn conversation, execution rails that validate tool calls, their arguments and results, and output rails that filter, edit or block responses. Whatever tool you use, the slots are the same. The OWASP Top 10 for LLM Applications 2025 maps onto them:

Input checks: injection, PII and topic scope

• Prompt injection detection: a classifier trained on known attacks, heuristics for telltale patterns such as hidden text and instruction-like phrasing in documents, or a model asked whether the text tries to give orders. These catch careless attacks. Paraphrases, other languages and encodings get through, so treat detection as one signal. The defense that holds is limiting what an injected instruction can make the system do, which is the job of the tool policy below. OWASP also recommends separating and labeling external content so the model can tell data from instructions. See prompt injection for the full picture.
• PII detection: decide which personal data may reach the model, the logs and any third-party API, then enforce it. Structured identifiers such as emails, card numbers and phone numbers fall to patterns and checksums; names, addresses and free-text health details need a named-entity model. The open-source Presidio project combines named-entity recognition, regular expressions, rule logic, checksums and surrounding context, and its documentation warns that automated detection can miss sensitive data and should be one protection among several. The actions are block, redact before sending, or replace values with tokens and restore them after the model answers.
• Topic scope: an assistant for one product does not need to write poems or give medical advice. A small classifier, or embedding similarity to a list of allowed intents, keeps it on task. NeMo Guardrails handles this with dialog rails. Off-topic requests are not always harmful, but out-of-scope chat is a common route into jailbreaks and a steady source of unwanted cost.
• Size and rate: cap input length, attachments and requests per user. That is the cheapest guardrail there is, and it covers OWASP's unbounded consumption risk.

Output checks: schema, grounding, toxicity and leakage

• Schema validation: parse the output and validate it against the schema the next step expects, then retry once or fail. Constrained decoding makes the syntax right, but not the values; see constrained decoding and structured outputs. OWASP's guidance on improper output handling is to treat the model like any other user, with zero trust: validate before output reaches backend functions, encode it for its destination, such as HTML-escaping for a page, and use parameterized queries for anything that touches a database.
• Grounding and citation checks: for an answer built on retrieved documents, check that every citation points to a document that was actually retrieved, that quoted spans really appear in it, and, with an entailment model or a judge, that the claims are supported. How to evaluate RAG covers the same measures offline.
• Toxicity and content policy: a safety classifier scores the response against categories such as violence or self-harm. The same model often checks the input too.
• PII and secret leakage: the model can repeat personal data from its context, or recite its system prompt. Scan outputs with the same detectors as inputs, and plant a unique canary string in the system prompt so a leak is an exact-match check.

A pattern-based redactor is the cheapest PII check. This one handles emails, international phone numbers, card numbers with a Luhn checksum, and IPv4 addresses. We ran it; the comment is its output:

import re

def luhn_ok(digits):
    nums = [int(c) for c in reversed(digits)]
    return (sum(nums[0::2]) + sum(sum(divmod(2 * n, 10)) for n in nums[1::2])) % 10 == 0

PATTERNS = {
    "EMAIL": re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b"),
    "CARD": re.compile(r"\b\d(?:[ -]?\d){12,18}\b"),
    "PHONE": re.compile(r"\+\d(?:[ ().-]?\d){7,14}\b"),  # international form only
    "IPV4": re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
}

def redact(text):
    for label, pattern in PATTERNS.items():
        def sub(m, label=label):
            if label == "CARD" and not luhn_ok(re.sub(r"\D", "", m.group())):
                return m.group()               # digits that fail the Luhn check: keep
            return f"<{label}>"
        text = pattern.sub(sub, text)
    return text

print(redact("Ana Kowalski, ana.k@example.org, +44 20 7946 0958, card 4111 1111 1111 1111, "
             "order 1234 5678 9012 3456, call 020 7946 0958, host 10.0.0.12"))
# Ana Kowalski, <EMAIL>, <PHONE>, card <CARD>, order 1234 5678 9012 3456, call 020 7946 0958, host <IPV4>

The output shows its limits as much as its strengths. The name gets through, because names need a named-entity model. The local-format phone number gets through, because the phone pattern only knows the international form, and widening it would start redacting order numbers and dates. The order number is kept only because it fails the Luhn check; a 16-digit number that happens to pass would be redacted as a card. The IPv4 pattern also accepts impossible addresses such as 999.1.1.1, and none of it catches an email address spelled out in words or a street address. Use patterns as a fast first layer under a model-based detector, never as the only one.

Tool-call policy: allow, deny or confirm

For an agent, the most consequential output is not text but a tool call: send this email, run this query, delete this file. OWASP calls the failure excessive agency. The control is a policy layer that sees every call before it runs and returns one of three decisions:

• Allow: the tool is on the list and its arguments pass validation, such as a schema, a path that stays inside one directory, a URL whose host is on an allowlist, or a query that only reads.
• Deny: the tool is not listed, the arguments fail, or a rate limit or per-run cap is hit. The denial goes back to the model as an ordinary tool result with a reason, so the agent can correct course instead of crashing.
• Confirm: the call is valid but consequential, such as sending, paying or deleting, so a person approves it first. No answer, an error or a timeout counts as no.

agent-guard is one example of such a layer, an open-source package. A policy lists tools as allow, deny or approve, denies unlisted tools by default, checks arguments against a JSON Schema subset and constraint helpers for paths, hosts and read-only SQL, applies rate limits and per-run caps, and writes an audit log with secrets redacted. Approvals fail closed: no approval function, a thrown error or any answer other than yes is a no. Its README is explicit about the boundary: the guard decides whether a call runs, and it does not isolate what the tool then does. Tools still need the least access that works, such as a read-only database role or a scoped token. AI agent guardrails covers least privilege and how to design approval steps people actually read.

Rules, classifier models and LLM judges

Every check is one of three kinds of mechanism, and they trade latency, cost and false positives differently:

Llama Guard, from Inan and colleagues (2023), shows where classifiers have gone: a Llama 2 7B model fine-tuned to classify both prompts and responses against a safety taxonomy, which can be swapped through instructions with zero-shot or few-shot prompting. The authors report that it matches or beats existing content moderation tools on the OpenAI Moderation Evaluation dataset and ToxicChat. LLM judges are more flexible still, but Zheng and colleagues (2023) documented position bias, verbosity bias, self-enhancement bias and limited reasoning in them, while also reporting that strong judges agreed with human preferences more than 80 percent of the time, about as often as humans agree with each other. A judge reads the same untrusted text as the model it checks, so it can be injected too.

The usual design layers them: rules first because they cost nothing, a classifier for the fuzzy categories, and a judge only for cases that are ambiguous or high-stakes. Measure false positives as carefully as misses. A guardrail that blocks one honest user in twenty is a broken product, and people route around guardrails they do not trust.

A guardrail pipeline, and when to fail closed

The control flow is simple. What matters is what happens when a check cannot answer. In this sketch, input checks run in parallel under a deadline, any failing verdict blocks, and a check that throws or times out also blocks:

type Verdict = { ok: true } | { ok: false; reason: string };
type Check = (text: string) => Promise<Verdict>;
const REFUSAL = 'Sorry, I can\'t help with that request.';

async function runChecks(checks: Check[], text: string, ms = 800): Promise<Verdict> {
  let cancel = () => {};
  const expired = new Promise<never>((_, reject) => {
    const timer = setTimeout(() => reject(new Error(`no verdict in ${ms} ms`)), ms);
    cancel = () => clearTimeout(timer);
  });
  try {
    const verdicts = await Promise.race([Promise.all(checks.map((check) => check(text))), expired]);
    return verdicts.find((v) => !v.ok) ?? { ok: true };
  } catch (err) {
    return { ok: false, reason: `check failed: ${(err as Error).message}` }; // fail closed
  } finally {
    cancel();
  }
}

export async function guarded(input: string, model: (prompt: string) => Promise<string>,
  checks: { input: Check[]; output: Check[] }, log: (event: object) => void): Promise<string> {
  const pre = await runChecks(checks.input, input);
  if (!pre.ok) { log({ stage: 'input', reason: pre.reason }); return REFUSAL; }
  const output = await model(input);
  const post = await runChecks(checks.output, output);
  if (!post.ok) { log({ stage: 'output', reason: post.reason }); return REFUSAL; }
  return output;
}

We ran it with stub checks: a clean request passed, an output containing a fake key was blocked at the output stage, and a check that slept past the deadline and one that threw were both logged and blocked. Errors from the model call itself propagate to the caller, which should handle them like any other upstream failure. One common speed-up runs the input checks in parallel with the model call and discards the answer if a check fails. That is only safe when the model call has no side effects, meaning no tool calls, and nothing is streamed to the user before the verdict.

Fail closed means a check that cannot answer counts as a block; fail open means it counts as a pass. Fail closed is right for anything with side effects, anything irreversible, and anything touching regulated or personal data. Fail open can be right for low-stakes checks, such as topic scope on a public help assistant, where an outage of the classifier should not take the whole product down. If a check fails open, log and alert on every instance, or it will quietly stay open. Decide this per check and write it down next to the check.

Guardrails on streaming output

Streaming and output checks pull in opposite directions: the user wants the first token now, and the check wants the whole answer. There are three options. Buffer the full response and check it once, which is simple and gives up the benefit of streaming. Stream first and retract if a check fails, which means the user has already seen the text. Or check as you go: hold back a window of recent text, check everything generated so far, and release only text that has passed.

NeMo Guardrails documents the same trade-off for its output rails. Its streaming settings set a chunk size, 200 tokens by default, that the rails check at a time, and a context size, 50 tokens by default, carried over between chunks so text that spans a boundary is still judged whole. With stream_first on, the default, the client gets each chunk before the rails check it; with it off, the rails run first and blocked content never reaches the client, at the price of added latency per chunk. Smaller chunks mean a faster first token and more rail runs. The holdback approach looks like this in miniature; we ran it, and the comments are its output:

import re

BLOCK = re.compile(r"sk-[A-Za-z0-9]{8,}")   # stand-in for any output check

def guarded_stream(chunks, holdback=24):
    """Release text only once it has been checked and is holdback chars old."""
    text, sent = "", 0
    for chunk in chunks:
        text += chunk
        if BLOCK.search(text):              # check everything so far, not just the chunk
            yield "\n(response withheld)"
            return
        safe_until = max(sent, len(text) - holdback)
        yield text[sent:safe_until]
        sent = safe_until
    yield text[sent:]                       # stream ended cleanly: flush the tail

leak = ["To call the API, ", "set the Authorization ", "header to s", "k-ABCD1234", "wxyz and retry."]
print(repr("".join(guarded_stream(leak))))
print(repr("".join(guarded_stream(["Hello ", "there, ", "how can I help?"]))))
# 'To call the API, set the A\n(response withheld)'
# 'Hello there, how can I help?'

In the first run the key arrives split across two chunks. A check on the newest chunk alone would miss it; the check on the accumulated text catches it, and nothing of the key was released. The holdback has to be at least as long as the longest thing a check needs to see. Running a classifier on every chunk is expensive, so run it every so many tokens on a sliding window, and once more on the full text at the end. When a stream is cut, the interface should replace the partial answer with the refusal, not leave half a sentence on screen.

Logging, review and red-teaming

Every guardrail decision is data. Log the stage, the check and its version, the verdict, the score, the latency and a redacted copy of the text, attached to the trace of the request; see LLM observability and tracing. Review a sample of blocks to find false positives and a sample of passes to find misses, and watch the block rate of each check. A sudden jump means either an attack or a broken check, and both need a person.

Then attack your own system. NIST AI 600-1, the Generative AI Profile of the voluntary AI Risk Management Framework, published in July 2024, defines AI red-teaming as a structured testing exercise to probe an AI system for flaws and vulnerabilities such as inaccurate, harmful or discriminatory outputs, and names pre-deployment testing and incident disclosure among its primary considerations. OWASP likewise recommends adversarial testing that treats the model as an untrusted user. In practice: keep a growing suite of attacks covering direct and indirect injection, PII extraction, jailbreaks and tool misuse, run it on every change of prompt, model or guardrail, and add every real incident as a new case. Frameworks like NIST's and the privacy laws that govern personal data shape what you must log and disclose; this post explains them and does not interpret them for your situation. This is not legal advice.

If you are putting an AI agent in front of real users and real tools, see our AI agent development service: the code lives in your GitHub from day one, AI-assisted work is reviewed by an engineer, and the founders are on the calls.