Announcing the Artificial Analysis Pronunciation Robustness benchmark, measuring how reliably Text to Speech models say challenging text correctly - Google Gemini 3.1 Flash TTS leads at 88.1%, followed closely by SpaceXAI TTS at 87.6% and ElevenLabs Eleven v3 at 85.6%
Existing Text to Speech (TTS) evaluations, including our TTS Arena, capture overall listener preference (e.g., how natural a voice sounds), and Word Error Rate (WER) checks whether the right words come out. Pronunciation Robustness adds a view of whether those words are said correctly (e.g., reading “St.” as “Saint” and “Street” in “St. Mary’s is on Church St.”) - this is critical for production voice agents, which need to get account details, names, currency amounts and more right to be trusted by users, at the low latency that conversational experiences demand.
Overview of Pronunciation Robustness
Each model generates audio for 454 sentences containing 701 target words or phrases, across four categories:
1. Contextually appropriate (words read differently depending on context, e.g., a wound that is bandaged vs. a bandage that is wound)
2. Expanding shorthand (numbers, dates, units and notation read out naturally,…
The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → tests 454 sentences covering 701 target words across four categories (contextual readings, expanding shorthand, exact sequences, standalone terms); a model is published only once 95% of targets have three or more approved listener judgments.
Hardest categories are expanding shorthand (62.4%) and preserving exact sequences (62.9%), trailing contextually-appropriate (86.2%) and standalone terms (86.1%) - scores were measured with no text normalization beyond each model's own default setting.
Preference and pronunciation accuracy diverge sharply: Sonic 3.6 ranks #1 on AA's Provider Voice Arena (1276 Elo) but only #11 on pronunciation robustness (74.5%), while Gemini 3.1 Flash TTS ranks #9 on the Arena (1201 Elo) yet #1 on pronunciation.
Category leaders split three ways: Gemini 3.1 Flash TTS leads contextually-appropriate and expanding-shorthand, SpaceXAI TTS leads preserving exact sequences, and Qwen-Audio-3.0-TTS-Plus leads standalone terms despite trailing on the overall leaderboard.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
For production voice agents, correctly pronouncing account numbers, abbreviations, and names matters as much as sounding natural; this benchmark gives engineers a way to pick TTS models on that specific, previously unmeasured failure mode.
Read the full post on X ↗
