Selecting a text-to-speech (TTS) voice – the voice someone speaking with an AI agent hears – sounds simple: generate a few samples, listen to them, and choose the one everyone likes.
We learned quickly that this approach does not work for production systems, especially in healthcare. A voice can sound warm in a demo while mispronouncing a medication, reading a ZIP code incorrectly, speaking too quickly, or introducing pauses that change the meaning of an instruction.
Selecting a text-to-speech (TTS) voice – the voice someone speaking with an AI agent hears – sounds simple: generate a few samples, listen to them, and choose the one everyone likes.
A good production voice is not merely pleasant. It must be understandable, accurate, reliable, contextually appropriate, and operationally fast.
Why voice evaluation is hard
Voice quality is multidimensional. A voice may sound natural but perform poorly when reciting numbers. Another may pronounce medications correctly but sound rushed or impersonal. A third may work well through headphones but become difficult to understand over a telephone codec.
The definition of “good” also depends on the use case. An advertising voice can be energetic and fast. A healthcare voice may need to be slower, calmer, and especially deliberate when reading medication names, dosages, dates, phone numbers, and instructions.
Automated evaluation introduces another complication. We transcribe generated speech with speech-to-text (STT) and compare it with the expected text, but an error can come from either system. The TTS voice may have pronounced something incorrectly, or the STT model may have misunderstood correct speech. Automated results therefore provide evidence, not absolute truth.
Qualities such as empathy, professionalism and appropriate urgency are even harder. Waveform statistics and transcription accuracy cannot reliably determine whether a voice sounds compassionate or reassuring. Those qualities still require structured human listening.
What we built
We created a provider-neutral voice benchmarking system called CareVoiceSelect. It treats a voice like a versioned software dependency rather than a subjective design choice.
For every candidate voice, the system:
- Generates the same collection of healthcare sentences.
- Saves every audio file for inspection.
- Transcribes the audio using STT and ASR
- Compares the transcriptions with the expected wording.
- Measures pronunciation, entity accuracy, pacing, pauses, latency, and audio quality.
- Applies strict production gates.
- Ranks candidates that meet the same eligibility level.
- Produces timestamped HTML, JSON, Markdown, and test artifacts.
- Provides a structured human PASS or FAIL review for qualitative traits.
The test pack includes ordinary sentences as well as intentionally difficult examples: medication names, dosages, appointment dates, confirmation codes, ZIP codes, urgent instructions, privacy-sensitive conversations, and emotionally difficult situations.
Most importantly, passing gates and ranking are separate concepts. A voice cannot compensate for an incorrect medication name by sounding more natural. A highly ranked voice may still be unsuitable for production if it fails a required safety check.
What teams should test
Transcript accuracy
Measure both word error rate and character error rate. WER catches missing, added, or substituted words, while CER can reveal smaller pronunciation and spelling differences.
Do not rely on either metric alone. A low average WER can hide a critical error such as changing “fifteen” to “fifty.”
Critical entities
Evaluate medications, dosages, units, dates, times, phone numbers, confirmation codes, and ZIP codes independently.
These should normally be release gates, not minor contributors to an average score. In a safety-sensitive workflow, one incorrect entity may matter more than dozens of correctly spoken filler words.
Pronunciation
Maintain a domain-specific pronunciation pack containing medication names, product names, abbreviations, clinicians’ names, and other specialized terms.
Listen to the original audio whenever STT reports a pronunciation failure. This distinguishes a genuine TTS problem from an ASR recognition error.
Pace and pauses
Measure words per minute, but also examine how the voice groups information. A reasonable average speed can still hide a rushed medication instruction or an unnatural pause in the middle of a phone number.
Use scenario-specific targets. Healthcare explanations often benefit from a slower pace than ordinary conversation.
Audio quality
Check for clipping, excessive silence, DC offset, unstable volume, truncation, and unexpected artifacts. Test the audio in the format used by production, including telephone codecs when applicable.
Latency
Measure complete-response latency as well as true streaming time to first audio. These answer different questions. A non-streaming WAV benchmark does not predict how quickly an interactive voice agent begins speaking.
Human communication quality
Use a consistent binary rubric for empathy, professionalism, clarity, naturalness, closure, privacy tone, cognitive load, and urgency calibration.
A reviewer should mark a scenario PASS only when every required dimension is appropriate. Otherwise, it should be FAIL. For consequential use cases, use at least two independent reviewers and document disagreements.
Regression stability
Pin the provider, voice ID, model snapshot, pronunciation dictionary, test pack, and evaluation configuration. Rerun the benchmark whenever any of them changes.
Store timestamped artifacts rather than replacing previous results. Voice APIs evolve, and a voice that passed last month should not be assumed to behave identically today.
The key lesson
There is no universally “best” voice, and a single score should never make the production decision.
The right voice is one that passes the non-negotiable requirements for its intended environment, performs consistently across difficult cases, and succeeds in structured human listening. Ranking should help compare eligible candidates, not hide failures.
Voice selection becomes much more reliable when we stop treating it as a demo-room preference and start treating it as a repeatable engineering, safety, and quality-assurance process.
