# AI defects: severity and bug report (cheat sheet)

> Classify, rate and report AI defects so they can be reproduced.

Source: https://conversationalaidesign.com/cheatsheets/qa-severity-and-bug-report.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## Defect types
- Ungrounded answer
- Policy violation
- Wrong tool or arguments
- Unsafe action (no confirmation)
- Over-refusal
- Handoff failure
- Language issue
- Conversation defect (forgets, loops)
- Leakage (prompt, data, credentials)
- Performance (latency, cost)

## Severity
| Level | Meaning |
|---|---|
| Critical | Harm, legal exposure, data leak, unsafe action |
| High | Wrong commitments, policy violations, broken core flows |
| Medium | Wrong or unhelpful answer with a safe fallback |
| Low | Tone, formatting, minor wording |

## Bug report fields
- Title, severity, type, reproduction rate (n of N runs)
- Environment, channel, language
- Versions: model, prompt, knowledge base
- Full transcript
- Retrieved passages and tool calls
- Expected behavior and spec ID

## Release gates
- Regression suite at required rates by severity
- No unresolved critical or high safety findings
- Coverage of every high-risk cell
- Latency, cost and accessibility within targets
- Canary rollout, monitoring and a kill switch

Related modules: A QA playbook for testing AI (https://conversationalaidesign.com/modules/16-qa-playbook.html)