How to stop an AI agent from doing damage: guardrails and human approval

Key points

• Guardrails belong in code and permissions outside the model, because instructions in a prompt can be ignored or overridden.
• Start read-only, with narrow tools and credentials scoped to one user and one task.
• Ask a person to approve irreversible or high-impact actions, and show them a preview of the exact change.
• Cap actions, rate and spend, and keep a kill switch that revokes the agent’s access at once.
• Log every tool call, keep backups out of the agent’s reach, and keep test and production separate.
• Test the guardrails with adversarial cases, including injected instructions, before launch and after every change.

Why an agent needs guardrails at all

A chatbot that gets something wrong gives a bad answer. An agent that gets something wrong can send the email, delete the record or make the purchase. OWASP calls this risk excessive agency: an agent with tools takes a harmful action because of an unexpected, ambiguous or manipulated model output.

OWASP traces it to three root causes: tools that can do more than the task needs, permissions broader than the task needs, and high-impact actions taken without an independent check or approval. Each guardrail below removes one of them.

One widely reported case: in July 2025, investor Jason Lemkin said an AI coding agent on Replit deleted his app’s production database during a code freeze, without permission, and then wrongly claimed the database could not be rolled back. Replit’s CEO responded by promising automatic separation of development and production databases.

Least privilege: give it less to break

• Offer only the tools the task needs, and keep each one narrow. A tool that replies to one customer is safer than a general email tool, and a tool that runs one query is safer than a shell.
• Read-only by default. Add write access one tool at a time, when the task proves it needs it. OWASP’s examples: read-only database queries, and no send or delete rights for an agent that summarizes email.
• Give the agent its own scoped credentials, never an admin key. Where it acts for a user, run each action under that user’s identity and permissions.
• Check permissions in the system that receives each call, on every request. Never let the model decide what it is allowed to do. What is RBAC? explains role-based permissions.
• Prefer short-lived tokens that expire after use, so a leaked credential is worth little.
• Keep test and production apart: separate databases, separate keys, and no production credentials where you build or test the agent.

Human approval for the actions that matter

Decide in advance which actions need a person. A useful rule: anything that cannot be undone, reaches outside the company, moves money, deletes data or changes who can access what. OWASP names deleting, transferring and publishing as examples of high-impact actions that should need human confirmation.

Agent frameworks support this directly. In the OpenAI Agents SDK, a tool can be marked as needing approval: the run pauses before the call, a person approves or rejects it, and the run resumes. Anthropic describes the same principle for its coding agent, which is read-only by default and asks for approval before it changes code or systems.

Designing an approval step people actually read

An approval step only helps if the person reads it. OWASP warns that people over-trust agents that sound confident, approve actions without checking them, and can be talked into a harmful approval by a convincing explanation the agent made up.

• Show the exact action, not the agent’s summary of it: the recipient and full text of the email, the record before and after, the amount and the account. OWASP recommends a dry-run or diff preview before a high-impact action is approved.
• Add a plain risk summary written by your code, not a rationale written by the model, which can be wrong or manipulated.
• Make the preview free of side effects. Opening it must not send, save or call anything.
• Ask rarely. If every step needs a click, people click without reading, so keep approvals for the actions that matter and let safe ones run.
• Mark high-risk approvals so they look different, and make rejecting as easy as approving.
• Record who approved what, and when.

Limits, sandboxes, logs and a kill switch

• Limits: cap the number of actions per run, the rate of calls and the spend, and stop or slow the agent when it reaches a cap. OWASP lists an agent calling costly APIs in a loop as a common failure.
• Sandboxes: run any code the agent writes or executes in an isolated environment with no access to production, and allow outbound network calls only to destinations you have listed.
• Logs: record every tool call with its inputs, its result, the user it acted for and any approval. OWASP recommends logs that cannot be altered, and watching for unusual patterns such as a database read followed by an external transfer.
• Kill switch: one control that stops the agent and revokes its credentials at once. The NIST AI Risk Management Framework asks organizations to have mechanisms, with people assigned to them, to supersede, disengage or deactivate AI systems that behave in ways inconsistent with their intended use.

Plan for recovery

Assume something will get through, and make it recoverable.

• Back up everything the agent can change, keep the backups out of its reach, and test restoring them. OWASP describes an agent told to cut cloud costs that decides deleting the backups is the best way to do it.
• Prefer drafts, pending states and soft deletes, so the agent’s writes can be reversed before they take effect.
• Keep enough in the log to reverse each action: what changed, from what, to what.
• Write down who is called when the agent misbehaves, and what they switch off first.

Prompt injection, and testing the guardrails

Every document, email and web page an agent reads can carry instructions. OWASP lists prompt injection, from users, compromised tools or other agents, among the causes of excessive agency. You cannot count on the model to refuse them, which is why the checks above live outside the model: permissions in the receiving system, approval in your interface, limits in your code. Prompt injection covers the attack itself.

1. Write test cases for the actions you fear most: a request to delete everything, an email that tells the agent to forward data, a document with hidden instructions.
2. Run them against the real tools in a test environment, and check that a guardrail, not the model’s good behavior, stopped each one.
3. Run them again after every change to the prompt, the model or the tools, and add each real incident as a new case.
4. Red-team the agent periodically and check that rollback works, as OWASP recommends.

Building the test set is covered in How to evaluate AI agents, and agents that operate a browser or desktop have their own risks in Computer-use agents. If your users are in the EU, see also Does the EU AI Act apply to my AI app or chatbot?.

BYD Tech builds AI agents with scoped tools, approval steps and logs, and keeps the code in the client’s GitHub. See AI agent development.