Voice assistants

Voice is different

A voice assistant removes the screen. That changes everything about the design:

• It's linear. You can't skim, scroll back or compare options at a glance.
• It's time-bound. Silence and delay are noticed immediately.
• Errors are audible. Mishearing a name or number is easy and embarrassing.
• It's public. People talk in cars, kitchens and offices, with noise and other people around.

Much of what you learned about conversation design applies, but the craft is closer to writing for the ear.

Two architectures

The two common ways to build a voice assistant.

Many production systems still use a cascade for control and auditability, and adopt speech-to-speech where naturalness matters most. Check your provider's current capabilities, since this area changes quickly.

The latency budget

Pauses feel long on the phone. A useful rule of thumb is to aim for the first audio within about a second of the user finishing, and to treat anything beyond a couple of seconds as a problem. The delay is the sum of several steps:

end-of-speech detection  +  speech recognition  +  model time-to-first-token
  +  tool calls (if any)  +  speech synthesis first audio  +  network

Ways to stay within budget:

• Stream every stage. Start speaking before the full reply exists.
• Use a fast model for conversation, and a bigger one only when needed (module 4).
• Acknowledge before you act: "Let me check that" while a tool runs, instead of dead air.
• Cache static prompts and context where possible.
• Tune end-of-speech detection: too eager cuts people off, too patient feels sluggish.

Voice UX patterns

Barge-in. People interrupt. Let them, and handle the assistant's own audio so it doesn't hear and respond to itself.

Keep lists short. Offer about three options, then "or I can read more."

Confirm critical details. Read back numbers and names in chunks.

Reprompt ladder. If there's no response or no match, don't repeat the same words. Rephrase, then give an example, then offer another way:

Always offer a non-voice path: keypad entry (DTMF), a link by text message, or a human.

Consent and disclosure. Say it's an automated assistant, and tell people if the call is recorded.

Writing for the ear

• Short sentences. One idea each.
• Say numbers the way people say them: "four one five" for a code, "twelve fifty" for a price, "November third" for a date.
• No markdown, URLs or emoji. Offer to send a link instead.
• Front-load the answer, then the detail.
• Use pauses. Many text-to-speech systems support markup such as SSML for breaks and emphasis.
• Vary phrasing. Reading the same reprompt twice sounds robotic.

Include the channel in your instructions: "You are speaking aloud. Do not use lists, markdown or links. Keep replies under 30 words unless asked for detail."

Multimodal experiences

For full journeys that cross channels, see module 21, and for speech across languages and dialects, see module 20.

Modern assistants can see images, read documents and share a screen, and many experiences mix voice with a screen. Use each for what it does best: voice for quick intent and confirmation, the screen for choices, comparisons, maps and anything the user should keep. Make sure each channel still works alone, since a person may be driving, may be deaf, or may be unable to speak.