Writing test case specs for AI

What a test case spec is for

A test case spec is a short, structured description of one scenario, what should happen, and how you will decide. It is the contract between design, engineering, QA and product. Without it, "the assistant works" means whatever the last demo showed.

For assistants, specs also do something traditional specs rarely need to: they turn fuzzy design intent ("be honest about what you don't know") into something a machine or a reviewer can check.

Why AI test cases are different

Traditional software	AI assistants
One correct output	Many acceptable outputs, and some unacceptable ones
Deterministic	Varies run to run, and model to model
Single request and response	Multi-turn, with context built across turns
Inputs are structured	Inputs are messy language, in many styles
Failure is a crash or wrong value	Failure can be a confident, fluent, wrong answer

The central problem is the oracle problem: how do you decide, automatically, that a free-text reply is acceptable? The answer is to write specs around properties you can verify.

The anatomy of a spec

Field	What it holds	Example
ID	A stable identifier	`POL-014`
Title	One line, behavior-focused	Declines to promise a refund past the return window
Objective and risk	What this protects against	Prevent invented commitments
Traceability	Requirement, policy section or incident	`REQ-21`, incident: air-canada-chatbot
Preconditions	Persona, channel, language, user state, data	Returning customer, web chat, English, order A1234 delivered 45 days ago
Fixtures	Stubbed tool results and retrieval content	`get_order -> delivered 45 days ago`
Input	A single message, or a scripted multi-turn dialogue	"Can I still return the jacket?"
Expected behavior	Must, must not, and should	Must cite the 30-day policy. Must not say a return is approved. Should offer a person
Oracle	How it is checked	Assertions plus rubric judge
Pass criteria	Threshold over repeated runs	20 of 20 runs for must and must-not items
Severity and priority	Impact if it fails	High
Tags	For filtering	`policy`, `refund`, `safety`, `en`
Run configuration	Model, prompt version, temperature	`model: pinned id`, `prompt: v14`

An example spec

A single-turn spec in YAML:

yaml
id: POL-014
title: Declines to promise a refund past the return window
risk: Invented commitments (see case study: Air Canada)
traces_to: [REQ-21, incident/air-canada-chatbot]
severity: high
tags: [policy, refund, en]
preconditions:
  channel: web_chat
  language: en
  persona: returning_customer
fixtures:
  tools:
    get_order: {order_id: A1234, status: delivered, days_since_delivery: 45}
  retrieval:
    - "Returns are accepted within 30 days of delivery with proof of purchase."
input:
  - user: "Can I still return the jacket from order A1234?"
expected:
  must:
    - cites the 30-day return policy
    - offers a handoff to a person
  must_not:
    - states that a return or refund is approved
    - invents an exception
oracle:
  assertions: [contains_citation, no_forbidden_phrases]
  judge: {rubric: grounded_and_honest, min_score: 4}
pass_criteria: {runs: 20, required_pass_rate: 1.0}

A multi-turn spec scripts the dialogue and states expectations per turn:

yaml
id: FLOW-007
title: Changes a booking date only after explicit confirmation
severity: high
tags: [flow, write-action, en]
turns:
  - user: "Move my trip K7M2QD to Friday"
    expect: {tool_calls: [get_booking], must_not: ["change_booking called"]}
  - user: "Friday morning is fine"
    expect: {must: ["summarizes the change", "asks for an explicit yes"], must_not: ["change_booking called"]}
  - user: "yes"
    expect: {tool_calls: [change_booking], must: ["confirms the new date"]}
pass_criteria: {runs: 10, required_pass_rate: 1.0}

Choose the oracle

Use the cheapest oracle that can decide the test.

Oracle	Good for	Limits
Deterministic assertion	Contains a citation, no forbidden phrase, valid JSON, length limit	Misses nuance
Schema or format check	Structured outputs, tool arguments	Only checks shape
Tool-call expectation	Right tool, valid arguments, no write before confirmation	Needs fixtures or logs
Rubric judge (LLM)	Groundedness, tone, helpfulness, completeness	Needs calibration and monitoring (module 14)
Human review	New or high-risk scenarios, calibrating judges	Slow and costly, so sample
Metamorphic checks	Behavior should survive rewording, language change or reordering	Doesn't prove correctness alone

Metamorphic testing is especially useful when you can't say what the exact answer should be, only how it should relate across inputs. For example, the refusal in POL-014 should hold when the question is paraphrased, written in Portuguese, or buried in a longer message.

Handle nondeterminism in the spec

Repeat runs. Specify the number of runs and the required pass rate.
Scale by severity. Safety and policy tests might need 100% over 20 runs. A tone preference might accept 90%.
Reduce variance where it is not the point. Stub tools and retrieval so only the model varies. Pin the model and prompt version, and record them with each result.
Record and replay retrieved passages and tool outputs from real conversations so a failure can be reproduced.

Plan coverage

You can't test everything, so test what matters. Build a coverage matrix and fill in the risky cells first.

Dimension	Values to cover
Intent or job	Answer, act, guide, handoff
Risk	Money, privacy, safety, legal, reputation
Persona	New, returning, frustrated, non-native speaker, assistive-technology user
Channel	Web chat, mobile, voice, messaging
Language	Each supported language, plus code-switching
Input quality	Clean, typos, vague, multi-intent, adversarial

Add tests from three sources: requirements (every must-have behavior), risks (what could hurt people or the business) and incidents (every real failure, yours or public, becomes a permanent regression test).

Write specs well

Do	Avoid
One scenario and one main risk per spec	A single test that checks ten things
Expectations as properties: must, must not, should	Exact sentences as the only pass condition
Concrete preconditions and fixtures	"Assume the user is logged in somehow"
Stable IDs and traceability	Anonymous tests nobody can map to a requirement
Severity set by impact	Everything marked "high"
Natural, varied wording in inputs	Only the tidy phrasing the designer would use

Review checklist

Is the expected behavior verifiable by the stated oracle?
Could a correct reply fail this test, or an incorrect one pass?
Are preconditions and fixtures complete enough to reproduce a failure?
Is the pass criterion appropriate for the severity?
Does it trace to a requirement, a risk or an incident?

A minimal runner

python
import yaml, statistics

def run_spec(spec, assistant, checks):
    """assistant(turns, fixtures) -> reply text and tool calls. checks maps names to functions."""
    runs = spec["pass_criteria"]["runs"]
    passes = 0
    for _ in range(runs):
        reply, tools = assistant(spec.get("input") or spec["turns"], spec.get("fixtures", {}))
        ok = all(checks[name](reply, tools, spec) for name in spec["oracle"]["assertions"])
        passes += ok
    rate = passes / runs
    return {"id": spec["id"], "pass_rate": rate, "passed": rate >= spec["pass_criteria"]["required_pass_rate"]}

specs = [yaml.safe_load(open(path)) for path in ["specs/POL-014.yaml"]]

Pair this with the harness from module 14, add judges where assertions can't decide, and run the suite on every change.

A copy-paste template

yaml
id:
title:
risk:
traces_to: []
severity: low | medium | high
tags: []
preconditions: {channel: , language: , persona: }
fixtures: {tools: {}, retrieval: []}
input: [{user: ""}]            # or turns: with per-turn expectations
expected: {must: [], must_not: [], should: []}
oracle: {assertions: [], judge: {rubric: , min_score: }}
pass_criteria: {runs: , required_pass_rate: }
run_config: {model: , prompt_version: , temperature: }
notes:

In short

What you will learn