Voice localization: STT and TTS across languages

Localization is more than translation

Translating the text of an assistant is the first ten percent. In voice, you must also adapt how people sound, what they say, and how the system should speak. The McDonald's drive-thru trial, which reportedly struggled with accents and dialects, shows the cost of treating speech as language-neutral.

Locale, not just language

Use locale tags such as pt-BR, pt-PT, es-MX, es-AR and en-IN (the format is defined in BCP 47). A locale carries differences in:

• Vocabulary: words for the same thing vary by region.
• Pronunciation and accent: recognition and voice models are often locale-specific.
• Formality and address: tu/você/o senhor, tú/usted/vos, du/Sie, tu/vous.
• Formats: dates (day first or month first), 12 or 24-hour time, decimal and thousands separators, currency, units, phone-number grouping.
• Conventions: politeness, turn-taking, tolerance for silence.

Decide which locales you support, and test each one as its own product.

Speech recognition (STT)

Evaluate honestly. Build a test set from real or representative speakers for each locale, including accents, ages, devices and noise. Measure word error rate (WER) overall and on the words that matter: names, numbers, intents. Compare across accents to find unfair gaps. Pair it with task success and escalation rates by locale (module 17).

Speech synthesis (TTS)

• Voice choice. Pick a voice for each locale, check it with native listeners, and keep a consistent persona.
• Text normalization. Numbers, currency, dates, abbreviations, ordinals and symbols must be expanded the way that language speaks them.
• Names and loanwords. Brand names, surnames and foreign terms often mispronounce. Control them explicitly.
• Pace and pauses. Languages and contexts differ. Read digits slowly in groups, and pause between options.
• Avoid fragment concatenation. "You have" + 2 + "message(s)" breaks plural and gender rules in many languages. Write complete messages for each case.

SSML and lexicons

SSML (Speech Synthesis Markup Language, a W3C standard) lets you control pronunciation and pacing. Pronunciation lexicons (PLS) let you define how specific words are pronounced and reuse that across messages. Support for individual tags varies by provider, so confirm in your provider's documentation.

<speak version="1.1" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="pt-BR">
  Sua reserva
  <say-as interpret-as="characters">K7M2QD</say-as>
  está confirmada para
  <say-as interpret-as="date" format="dmy">04/11/2026</say-as>.
  <break time="400ms"/>
  O voo sai do
  <phoneme alphabet="ipa" ph="ɡwaˈɾulʊs">Guarulhos</phoneme>
  às
  <say-as interpret-as="time" format="hms24">14:30</say-as>.
</speak>

LOCALE = {
    "pt-BR": {"decimal": ",", "thousands": ".", "currency": "R$ {amount}", "formal": "o senhor"},
    "en-US": {"decimal": ".", "thousands": ",", "currency": "${amount}", "formal": "you"},
    "de-DE": {"decimal": ",", "thousands": ".", "currency": "{amount} Euro", "formal": "Sie"},
}

def format_amount(value: float, locale: str) -> str:
    cfg = LOCALE[locale]
    s = f"{value:,.2f}"                      # 1,234.56
    s = s.replace(",", "§").replace(".", cfg["decimal"]).replace("§", cfg["thousands"])
    return cfg["currency"].format(amount=s)

print(format_amount(1234.5, "pt-BR"))        # R$ 1.234,50

In production, use a maintained internationalization library (for example the Unicode CLDR data and ICU) rather than hand-rolled formatting.

Content that localizes correctly

• Plurals and agreement. Use message formats with plural rules per language (ICU MessageFormat is a common standard) instead of concatenation.
• Formality. Choose and document the register for each locale, and apply it in prompts and messages.
• Prompts across languages. Many teams keep the system prompt in one language and instruct the model to reply in the user's language. Others maintain translated prompts. Test both on each locale, because quality, tone and refusals differ (module 5).
• Terminology. Keep a glossary of product terms, with approved translations and "do not translate" items such as tool names and brand names.
• Examples. Few-shot examples should reflect local phrasing, not translated English.

Language switching and code-switching

People mix languages, especially for product names, technical terms and quotations. Plan for it:

1. Detect a switch in the user's language.
2. Confirm gently when it matters: "Would you prefer to continue in English?"
3. Keep the context across the switch. Don't restart the conversation.
4. Allow mixed input such as local-language sentences with English product names, and make sure recognition vocabulary includes both.
5. Respond in the language they chose, not necessarily the one they used last.

Illustrative gotchas

These are examples, not a complete list, and they are why native-speaker review is essential.

Test per locale

• Native-speaker review of scripts, voices and fallbacks.
• Locale-tagged specs (module 15): language, locale and formats as preconditions, and parallel test cases across locales.
• Audio tests with diverse speakers, devices and noise.
• Pseudo-localization of text to catch layout and concatenation problems early.
• Dashboards segmented by locale: recognition errors, fallback rate, escalation rate, satisfaction (module 17).
• A fairness check: performance gaps between accents and groups, with owners and deadlines to close them.