Pronouncing Hard Words: Phoneme Adaptation of a Pretrained Conversational Text to Speech Model

Voice agents built as cascaded automatic speech recognition (ASR) → large language model (LLM) → text-to-speech (TTS) pipelines increasingly rely on conversational TTS: streaming, context condi- tioned models that adapt delivery to dialogue history. Such models are natural and expressive but offer little explicit control over how rare, trust-critical drug names are pronounced, because these enti- ties are too infrequent, and their vocabulary too fast-growing to be covered by any amount of pre-, mid-, or post-training. We retrofit phoneme controllability onto a pretrained conversational TTS model avoiding any pre/mid-training, using an ARPAbet vocabulary exten- sion, continued phoneme-augmented supervised fine-tuning (SFT), and a training-free inference-time acoustic-biasing step. On a 7,500- clip clinical drug-pronunciation evaluation, our model raises strict Gemini-judged correctness from 54.6% (no phoneme augmenta- tion) to 90.7%, outperforming every conversational and stateless TTS baseline on the automatic metric. Because no training stage sees a drug name, the entire evaluation is out-of-domain, so the gains reflect learned pronunciation rather than memorization. In a blinded, randomized human study, adult listeners judge 84% of our clips correct, the highest across all the baselines. Across all 250 matched clips, Gemini agrees with the human majority on 84.8% of binary labels (κ=0.57).

Audio Samples

We evaluate pronunciation accuracy on three held-out drug names. For each drug, we show a human reference recording followed by five generations from each model. Samples are scored 1–3 by a Gemini judge (3 = correct).

Gemini-as-Judge Prompts

Drug-Pronunciation Judge

The drug-pronunciation eval calls Gemini-3.1-Pro at T = 0 with two audio inputs (AUDIO 1 = reference recording of the drug name spoken in isolation; AUDIO 2 = the TTS-generated clip) and the prompt below. The IPA block is shown only as descriptive context for error annotations and is explicitly excluded from the scoring decision.

You are evaluating whether a TTS system correctly pronounced a medical/drug term. Focus ONLY on the pronunciation of the target term. Ignore the surrounding sentence quality, speaker voice, pacing, and expressiveness.

Target term: {drug_name}
Reference IPA (for error description only -- do NOT use this to determine your score): {ipa}
AUDIO 1: Reference recording of "{drug_name}" spoken in isolation.
AUDIO 2: TTS synthesis of the sentence: "{sentence}"

## Scoring instructions

Base your score SOLELY on how the pronunciation in AUDIO 2 compares to AUDIO 1. Do NOT score against the IPA text -- the reference audio is the only ground truth.

Rate the pronunciation of "{drug_name}" in AUDIO 2:
  3 = Correct: matches the reference audio clearly and correctly
  2 = Flawed but recognizable: identifiable as the same drug, but has audible errors vs the reference
  1 = Wrong or unintelligible: sounds like a different word, is garbled, or is skipped entirely

Output structured JSON:
  - If 3: {"score": 3}
  - If 2/1: include error_type, heard_as (phonetic description), and confidence (high|medium|low).

The corpus-level strict correct rate (% of clips with score=3) is reported as mean ± 5-run σ from a 5-generation partition. The same prompt is applied verbatim to the Drug-OOD split and to every cross-system row.

ARPAbet Vocabulary Extension (131 tokens)

Controllability is exposed via a fixed ARPAbet phone inventory registered as atomic special tokens. A pronunciation span is wrapped as <|arpa_start|>…<|arpa_end|> with one token per phone (e.g. <|arpa_HH|>). The inventory is a CMU-ARPAbet + TIMIT superset: 129 phones plus the two span markers = 131 tokens.