Multimodal journeys

From channels to journeys

Customers don't think in channels. They think in tasks: "file a claim," "change my trip," "fix my bill." A multimodal journey is a task that spans voice, chat, apps, web pages, images and documents. The craft is in the seams: how context, identity and trust move when the modality changes.

Choose the right modality for each step

Map the journey

Draw a swimlane diagram with one lane per channel and mark every handoff. For each seam, define:

1. Why the switch happens, and how you explain it to the user.
2. What travels: a summary of the task, collected values, the identity level reached and consent given.
3. How it travels: a signed, expiring reference to context on your server, not a pile of data in a link.
4. How they return: a way back, and what happens if they never complete the step.
5. What if it fails: a fallback (module 11).

An insurance claim across five touchpoints. The seams are where journeys break.

Identity, security and consent across channels

• Step-up authentication. Verify identity at the level the next action needs, and re-verify when moving to a riskier action.
• Short-lived, single-purpose links and one-time codes. Never place personal data in a URL.
• Channel-appropriate content. Don't send sensitive details over channels that are easy to intercept.
• Consent per channel and purpose. Permission to talk on the phone isn't permission to text (module 13).
• Data minimization. Pass references, not records.

A signed, expiring resume token is a simple building block:

import base64, hashlib, hmac, json, time

SECRET = b"load-from-your-secret-manager"          # never hard-code in production

def make_resume_token(journey_id: str, step: str, ttl_seconds: int = 1800) -> str:
    payload = {"j": journey_id, "s": step, "exp": int(time.time()) + ttl_seconds}
    body = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode().rstrip("=")
    sig = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
    return f"{body}.{sig}"

def read_resume_token(token: str) -> dict | None:
    try:
        body, sig = token.split(".")
        expected = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
        if not hmac.compare_digest(sig, expected):
            return None
        payload = json.loads(base64.urlsafe_b64decode(body + "=" * (-len(body) % 4)))
        return payload if payload["exp"] > time.time() else None
    except Exception:
        return None

The token only names the journey and step. The actual task state lives on your server, behind authentication.

Images and documents

Modern models can interpret images and documents, which opens strong journeys and new risks.

Images - Good uses: damage and condition evidence, product identification, reading labels and screens, location context. - Failure modes: blur, poor light, the wrong object, misreading text, over-confident descriptions. - Design: guide the capture ("show the whole bumper, in daylight"), check quality before analysis, say what the assistant sees and ask the user to confirm, and keep humans in the loop for decisions with consequences. - Privacy: faces, documents and background details, plus location data embedded in files. Collect only what you need, tell people how it's used and how long it's kept.

Documents - Extract fields, then show them and ask for confirmation or correction before acting. - Validate against rules (dates in range, totals that add up, required fields present). - Treat uploaded files as untrusted input: they can contain hidden instructions (module 18).

Screen and rich responses

Chat doesn't have to be text only. Cards, buttons, forms and carousels can make choices clearer. Emerging standards extend protocols such as MCP with ways for tools to return interface components (the 2026-07-28 MCP specification formalized extensions including MCP Apps). Support varies by client, so design a plain-text equivalent for every rich response (module 8).

Architecture

One journey state, many renderers.

Keep a journey ID that follows the customer across channels, a shared state store for the task, and per-channel rendering so the same decision appears as speech, a message or a screen as appropriate.

Test cross-channel journeys

• End-to-end scenarios that cross channels, with time gaps between steps.
• Resume tests: expired links, a second device, a different channel, a returning user.
• Failure injection: upload fails, link never opened, call drops mid-step (module 11).
• Security tests: reused or tampered tokens, wrong-person access.
• Accessibility: every step has an alternative for people who can't see, hear or speak.
• Analytics keyed by journey ID: drop-off at each seam, time between steps, and completion by path (module 17).