AI defects: severity and bug report
Classify, rate and report AI defects so they can be reproduced.
QADeveloper
Defect types
- Ungrounded answer
- Policy violation
- Wrong tool or arguments
- Unsafe action (no confirmation)
- Over-refusal
- Handoff failure
- Language issue
- Conversation defect (forgets, loops)
- Leakage (prompt, data, credentials)
- Performance (latency, cost)
Severity
| Level | Meaning |
|---|---|
| Critical | Harm, legal exposure, data leak, unsafe action |
| High | Wrong commitments, policy violations, broken core flows |
| Medium | Wrong or unhelpful answer with a safe fallback |
| Low | Tone, formatting, minor wording |
Bug report fields
- Title, severity, type, reproduction rate (n of N runs)
- Environment, channel, language
- Versions: model, prompt, knowledge base
- Full transcript
- Retrieved passages and tool calls
- Expected behavior and spec ID
Release gates
- Regression suite at required rates by severity
- No unresolved critical or high safety findings
- Coverage of every high-risk cell
- Latency, cost and accessibility within targets
- Canary rollout, monitoring and a kill switch