Guardrails for AI assistants | Conversational AI Design

What a guardrail is

A guardrail is a control that keeps an assistant inside acceptable behavior: what it may accept, retrieve, say and do. The key distinction is between asking and enforcing.

A prompt instruction asks the model to behave. It is useful but never a guarantee.
A guardrail in code checks or constrains behavior outside the model, so a failure of the model can't become a failure of the system.

Both matter, but anything with serious consequences needs the second kind. The DPD and Replit case studies are two sides of the same point: a rule written in prose was not enforced.

Guardrails at every layer

User inputInput guardrailsRetrieval guardrailsModel and promptTool guardrailsOutput guardrailsUser

Controls belong at each boundary, with conversation-level limits across the whole exchange.

Layer	What it controls	Examples
Input	What reaches the model	Personal-data detection and redaction, language detection, off-topic and abuse classifiers, injection heuristics, length and rate limits
Retrieval	What the model may read	Access-control filtering, source allowlists, relevance thresholds, current-version filters
Prompt	The model's role and rules	Scope, refusal style, instructions to ignore commands inside documents (helpful, not sufficient)
Tool	What the model may do	Tool allowlists, schema validation, per-user credentials, spending and rate limits, confirmation before writes, dry runs, separate environments
Output	What the user receives	Profanity and toxicity checks, policy classifiers, personal-data redaction, citation checks, format validation, forbidden promises and claims
Conversation	The whole exchange	Turn limits, loop detection, cost caps, escalation triggers, a kill switch

Rules, classifiers or model judges?

Method	Strengths	Weaknesses	Use for
Deterministic rules (regex, allowlists, schemas)	Fast, exact, explainable	Brittle, easy to evade, misses nuance	Hard limits: forbidden phrases, formats, permissions, amounts
Trained classifiers	Flexible, fast	Errors in both directions, needs tuning	Toxicity, topic, intent, injection patterns
LLM judges	Nuanced, can follow a rubric	Slower, costlier, can be fooled	Groundedness, tone, subtle policy checks, sampling reviews
Humans	Judgment	Slow, expensive	Approvals for consequential actions, audits

Layer them: cheap deterministic checks first, then classifiers, then a judge for what remains, with a human for the highest stakes.

Policy as data, enforcement as code

Write the rules in one place so designers, legal and engineers can read and review them, and let code enforce them.

yaml
policy_version: 7
rules:
  - id: no-refund-promises
    applies_to: output
    match: ["refund (has been|is) (approved|processed)", "I('ve| have) refunded"]
    action: rewrite        # replace with the approved wording below
    rewrite_with: "I can't confirm a refund myself. I can connect you with an agent."
    severity: high
  - id: no-profanity
    applies_to: output
    action: block
    severity: high
  - id: pii-in-input
    applies_to: input
    detect: [credit_card, national_id]
    action: redact
    severity: medium
  - id: write-needs-confirmation
    applies_to: tool
    tools: [cancel_booking, create_return_label]
    action: require_confirmation
    severity: high

python
import re

def check_output(text: str, rules: list[dict]) -> tuple[str, list[str]]:
    """Return (possibly rewritten text, ids of rules that fired)."""
    fired = []
    for rule in rules:
        if rule["applies_to"] != "output":
            continue
        if rule["id"] == "no-profanity" and is_profane(text):     # your classifier
            return "I'm sorry, I can't help with that. Is there anything about your order I can do?", [rule["id"]]
        for pattern in rule.get("match", []):
            if re.search(pattern, text, flags=re.IGNORECASE):
                fired.append(rule["id"])
                if rule["action"] == "rewrite":
                    return rule["rewrite_with"], fired
    return text, fired

ALLOWED_TOOLS = {"get_order", "get_booking", "create_return_label"}   # least privilege
NEEDS_CONFIRMATION = {"create_return_label"}

def guard_tool_call(name: str, args: dict, confirmed: bool) -> dict:
    if name not in ALLOWED_TOOLS:
        return {"allowed": False, "reason": "tool not permitted"}
    if name in NEEDS_CONFIRMATION and not confirmed:
        return {"allowed": False, "reason": "needs explicit user confirmation"}
    return {"allowed": True}

Log every rule that fires with the conversation ID and policy version. Those logs are your best source of new test cases and your evidence for audits.

Capability limits beat instructions

The strongest guardrails remove the ability to do harm:

Separate environments. An agent working on development resources shouldn't hold credentials for production. After the Replit incident, the company pledged automatic separation of development and production databases.
Least privilege and per-user credentials. The assistant can only do what that user could do.
Read-only by default. Writes need a separate tool, a confirmation and, for high impact, human approval.
Budgets. Cap steps, tokens, spend and time per conversation.
Dry runs and undo. Show what would happen, and make consequential actions reversible where possible.

Balance false positives and false negatives

Every guardrail makes two kinds of mistakes: letting bad things through (false negatives) and blocking good things (false positives). Over-blocking is a product failure too: customers who are refused legitimate help leave.

Build two test sets: harmful or out-of-policy cases, and legitimate but tricky ones.
Measure catch rate and false-positive rate for each rule.
Tune thresholds against both, per severity.
Review blocked conversations regularly, and add them to the sets.

Latency, streaming and voice

Guardrails add time. Run independent checks in parallel, keep blocking checks fast, and move slower reviews to asynchronous sampling. When streaming text or speech, buffer a sentence before releasing it so an output check can still stop a bad reply. In voice, a late block is audible, so design for it (module 19).

Test and monitor guardrails

Adversarial suites: injection attempts, tone attacks, extraction attempts, jailbreak patterns (module 16).
Regression: every bypass found becomes a permanent test.
Monitoring: rule-fire rates, spikes after releases, blocked-then-abandoned conversations.
Versioning: policy files are versioned and change-controlled like code.

Libraries and managed services exist for content filtering and policy checks (for example NeMo Guardrails and Guardrails AI, or your cloud provider's filters). Evaluate them against your own test sets, and keep your policy readable and portable.

Guardrails

In short

What you will learn