# Safety, security and ethics

> The risks that matter for assistants: prompt injection, excessive agency, data exposure, honesty and accountability, mapped to the OWASP Top 10 for LLM applications.

Source: https://conversationalaidesign.com/modules/18-safety.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## What you will learn
- Use the OWASP Top 10 for LLM Applications (2025) as a risk map
- Explain direct and indirect prompt injection and apply layered defenses
- Limit what an assistant can do with permissions and confirmations
- Design for privacy, honesty, accessibility and accountability
- Run a pre-launch safety review


## A risk map: the OWASP Top 10 for LLM applications

The OWASP GenAI Security Project maintains a widely used list of the most critical risks in applications built on language models. The 2025 edition lists:

| ID | Risk | In one sentence |
|---|---|---|
| LLM01 | Prompt injection | Attacker-controlled text changes the model's behavior |
| LLM02 | Sensitive information disclosure | The model reveals data it should not |
| LLM03 | Supply chain | Compromised models, adapters, datasets or dependencies |
| LLM04 | Data and model poisoning | Tampered training or fine-tuning data |
| LLM05 | Improper output handling | Model output used unsafely by downstream code |
| LLM06 | Excessive agency | The model can take damaging actions in connected systems |
| LLM07 | System prompt leakage | Hidden instructions are extracted |
| LLM08 | Vector and embedding weaknesses | Attacks on retrieval stores and embeddings (RAG) |
| LLM09 | Misinformation | Confident, false or misleading output |
| LLM10 | Unbounded consumption | Runaway usage, denial of service and cost attacks |

Use it as a checklist for design reviews. Prompt injection is first for a reason.

## Prompt injection

An LLM can't reliably separate *instructions* from *data*. Two forms matter:

- **Direct:** the user tells the model to ignore its rules.
- **Indirect:** instructions hidden in content the assistant reads, such as a web page, an email, a PDF, a retrieved passage or a tool description.

There is no complete fix, so defend in layers:

1. **Limit capability.** Give tools the least privilege and the narrowest scope. A hijacked model with no powerful tools can do little.
2. **Confirm consequential actions** with a human, showing exactly what will happen.
3. **Treat external content as data.** Tell the model to ignore instructions inside sources, and mark where untrusted text begins and ends. This helps, but don't rely on it alone.
4. **Separate duties.** The component that reads untrusted content shouldn't also hold sensitive data and the ability to send it elsewhere. When one system has private data, untrusted input and an outbound channel, assume an attacker can combine them.
5. **Validate outputs.** Check tool arguments, links and formats in code before acting.
6. **Monitor and test.** Red-team with injection attempts through every input channel, and log suspicious patterns.

## Excessive agency

The more an assistant can *do*, the more damage a mistake or attack can cause.

- Grant only the tools a task needs, with read-only access where possible.
- Put spending, deletion and messaging behind confirmation.
- Cap steps, time and cost per conversation.
- Use per-user credentials, so the assistant can only touch what that user could.
- Keep audit logs of every action.

## Privacy and data

- **Collect and send the minimum.** Don't put full records into prompts when a few fields will do.
- **Redact** personal data before logging and before sending it to third parties where possible.
- **Know the data flow:** where prompts go, how long they are kept, whether they are used for training, which region processes them.
- **Memory needs consent.** Let people see and delete what is remembered.
- **Retention limits** for transcripts, and access controls on who can read them.

## Honesty, disclosure and accountability

- **Say it's an AI.** Don't present a bot as a person.
- **Offer a human,** especially for complaints, money and distress.
- **Don't promise what you won't honor.** Assistants that invent discounts or policies create real obligations.
- **Answer from authority.** Policy questions should come from approved sources (module 6), with a clear not-found path.
- **Own the outcome.** In *Moffatt v. Air Canada* (2024), a British Columbia tribunal found the airline responsible for incorrect advice from its chatbot and rejected the argument that the chatbot was a separate entity. Assume you will be held accountable for what your assistant says.

## Fairness, vulnerable users and accessibility

Test with different dialects, languages and writing styles. Provide accessible alternatives and plain language. Decide in advance how the assistant responds to people in crisis: acknowledge, avoid giving risky advice, and route to appropriate human or emergency resources. Medical, legal and financial topics need conservative scopes and clear handoffs.

## Governance

- **Name an owner** for the assistant's behavior and a process for incidents.
- **Keep a change log** of prompts, models, tools and knowledge.
- **Review before launch** with design, engineering, legal and support.
- **Check the rules that apply to you.** Several jurisdictions have disclosure, consumer-protection and data-protection requirements for automated systems. Ask qualified counsel rather than assuming.

> **Pre-launch safety checklist**
> - We have mapped our design to each OWASP LLM risk.
> - Tools use least privilege; write actions require confirmation.
> - Retrieved content and tool results are treated as untrusted.
> - There is a not-found path and a human handoff.
> - The assistant discloses that it is an AI.
> - Logs are redacted and retention is limited.
> - We have an adversarial test set and a kill switch.


## Key takeaways
- There is no single fix for prompt injection. Layer your defenses and limit what the model can do.
- The more authority you give an assistant, the more it must be constrained and monitored.
- Disclose that it is an AI, offer a human, and don't let it promise what you won't honor.
- A company can be held responsible for what its assistant says.

## Sources
- OWASP GenAI Security Project: Top 10 for LLM Applications: https://genai.owasp.org
- Moffatt v. Air Canada, 2024 BCCRT 149 (summary by Foster and Company): https://fosterandcompany.com/?p=5709
- AI Incident Database: https://incidentdatabase.ai
- Case study: Air Canada: https://conversationalaidesign.com/case-studies/air-canada-chatbot.html