# Orchestration: state machines, agents and multi-agent systems

> Choose the right architecture: deterministic flows, a single tool-using agent or several cooperating agents, and how to combine them in a hybrid design.

Source: https://conversationalaidesign.com/modules/09-orchestration.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## What you will learn
- Compare state machines, single agents and multi-agent systems
- Apply a decision framework instead of defaulting to agents
- Build a hybrid flow with a deterministic skeleton and LLM at the joints
- Route requests with confidence thresholds
- Recognize the main agent frameworks and what is durable beyond them


## Three architectures

Everyone wants "an agent." The better question is how much freedom the model should have. There are three common answers.

- State machine: you control the path → Single agent: the model picks the next step → Multi-agent: several specialized agents coordinate

### 1. State machine (flow)

You define explicit **states** (collect order number, confirm item, offer options, finish) and **transitions** between them. The LLM may understand messy input and write natural replies, but your code decides what happens next.

*Strengths:* predictable, testable, auditable. *Limits:* rigid, you must anticipate paths.

### 2. Single agent with tools

One model runs in a loop: think, call a tool, read the result, repeat until done. It chooses the steps.

*Strengths:* flexible, handles unforeseen requests. *Limits:* less predictable, harder to guarantee behavior, can loop or wander.

### 3. Multi-agent

Several agents, each with its own instructions, tools and context, coordinated by a supervisor or by handoffs. Examples: a triage agent that routes to billing, technical and sales agents.

*Strengths:* separation of concerns, different permissions per agent, parallel work, smaller contexts. *Limits:* more cost and latency, coordination bugs, harder debugging.

| | State machine | Single agent | Multi-agent |
|---|---|---|---|
| Control | Highest | Medium | Lowest |
| Flexibility | Low | High | Highest |
| Cost and latency | Lowest | Medium | Highest |
| Testability | Easiest | Moderate | Hardest |
| Best for | Regulated, well-defined processes | Open tasks with tools | Distinct domains, permissions or parallel work |

## A decision framework

Ask in order and stop at the first yes:

1. **Is the process well defined and the stakes high?** (payments, identity, legal commitments) Use a **state machine**.
2. **Are the steps unknowable in advance, but the tools few?** Use a **single agent** with tight tools and limits.
3. **Do you need different permissions, knowledge domains or parallel work that one context can't hold?** Consider **multi-agent**.
4. **Is it just questions about content?** Plain **RAG** (module 6) may be enough, no agent needed.

> **The simplest thing that works**
> Teams often reach for multi-agent because it sounds powerful. Start simple, measure where it breaks, and add structure only for a measured reason.

## The hybrid pattern: a deterministic skeleton with the LLM at the joints

The most robust designs let code own the *what* and the model own the *language*.

```python
from enum import Enum, auto

class S(Enum):
    ASK_BOOKING = auto(); ASK_DATE = auto(); CONFIRM = auto(); DONE = auto(); HANDOFF = auto()

def step(state, user_text, ctx, llm_extract, tools):
    """One turn. Code decides the transition; the LLM only extracts and phrases."""
    found = llm_extract(user_text)                       # {"booking_code": ..., "new_date": ...}
    ctx.update({k: v for k, v in found.items() if v})

    if state == S.ASK_BOOKING:
        if not ctx.get("booking_code"):
            ctx["misses"] = ctx.get("misses", 0) + 1
            return (S.HANDOFF, "I'm having trouble. Let me connect you to a person.") if ctx["misses"] >= 2 \
                   else (S.ASK_BOOKING, "What's your six-character booking code?")
        ctx["fare"] = tools["get_booking"](ctx["booking_code"])      # grounded in your system
        return S.ASK_DATE, f"Found it. Which date would you like instead?"

    if state == S.ASK_DATE:
        if not ctx.get("new_date"):
            return S.ASK_DATE, "Which date would you like?"
        return S.CONFIRM, f"Move your trip to {ctx['new_date']}? Please reply yes to confirm."

    if state == S.CONFIRM:
        if user_text.strip().lower() in {"yes", "y", "confirm"}:      # explicit confirmation gate
            tools["change_booking"](ctx["booking_code"], ctx["new_date"])
            return S.DONE, "Done. Your trip has been moved."
        return S.ASK_DATE, "No problem. Which date would you like?"
    return state, "How else can I help?"
```

Notice the guard rails: a **miss counter** that triggers handoff, a **confirmation gate** before the write action, and tool calls that come from code, not from the model's imagination. You could replace `llm_extract` with a small model, and swap the reply strings for model-generated text constrained by the state.

## Routing with confidence

A **router** classifies each request and sends it down the right path. This replaces the intent-platform-plus-LLM pattern from the first edition of this course with a version that works with any stack.

```python
def route(message, history):
    result = classify(message, history)        # returns {"route": "...", "confidence": 0.0-1.0}
    if result["confidence"] < 0.5:
        return "clarify"                        # ask a question rather than guess
    if result["route"] == "faq":
        return "rag"                            # grounded answer (module 6)
    if result["route"] in {"change_booking", "cancel_booking"}:
        return "flow"                           # state machine for transactions
    if result["route"] == "open_request":
        return "agent"                          # tools with limits
    return "human"
```

Tune the thresholds with your evaluation set, and log every routing decision so you can see where it goes wrong.

## When multi-agent is worth it

Good reasons: different **permissions** (the billing agent can issue refunds, the FAQ agent cannot); **separate knowledge domains** that overflow one context; **parallel** subtasks; independent teams owning separate agents (where A2A helps).

Poor reasons: it looks sophisticated; the prompt got long (try splitting the *prompt* first).

If you do build one, keep the number of agents small, make handoffs explicit and logged, share state through your own store rather than agents chatting freely, set budgets for steps and cost, and test each agent alone before testing them together.

## Memory and state

- **Conversation memory:** what you send each turn. Summarize long histories instead of replaying everything.
- **Session state:** structured facts (booking code, chosen date) in your own store, not only in the transcript.
- **Long-term memory:** preferences across sessions. Collect with consent, let users view and delete it, and keep it minimal.

## The framework landscape (October 2026)

Every major lab and cloud now ships an agent toolkit, and several independent ones are popular. As of this writing, commonly compared options include LangGraph, the OpenAI Agents SDK, the Claude Agent SDK, Google's Agent Development Kit, Microsoft Agent Framework (which merged AutoGen and Semantic Kernel), CrewAI, Pydantic AI, Mastra and LlamaIndex. They fall into a few groups: graph engines for stateful workflows, vendor SDKs tuned for one model family, and managed runtimes.

Don't pick one because it is trending. Pick by language and team, by how much durability and tracing you need, and by how committed you are to one vendor. And remember the point of this module: the **patterns** (states, tools, routing, handoffs, budgets, tests) outlive any framework.


## Key takeaways
- Start with the simplest architecture that meets the need.
- Use state machines where the process is known and the stakes are high.
- Use agents where steps can't be planned in advance. Use multi-agent only for a clear reason.
- A hybrid design (deterministic skeleton, LLM at the joints) is the most common good answer.
- Frameworks change; patterns persist.

## Sources
- Choosing an agent framework in late 2026 (Developers Digest): https://www.developersdigest.tech/blog/choosing-an-agent-framework-late-2026
- The 8 best AI agent frameworks in 2026 (Dupple): https://dupple.com/learn/best-ai-agent-frameworks
- A2A protocol: https://a2a-protocol.org