New benchmark tests whether TTS models pronounce tricky text correctly
- Source
- x.com
- Date
Announcing the Artificial Analysis Pronunciation Robustness benchmark, measuring how reliably Text to Speech models say challenging text correctly - Google Gemini 3.1 Flash TTS leads at 88.1%, followed closely by SpaceXAI TTS at 87.6% and ElevenLabs Eleven v3 at 85.6% Existing Text to Speech (TTS) evaluations, including our TTS Arena, capture overall listener preference (e.g., how natural a voice sounds), and Word Error Rate (WER) checks whether the right words come out. Pronunciation Robustness adds a view of whether those words are said correctly (e.g., reading “St.” as “Saint” and “Street” in “St. Mary’s is on Church St.”) - this is critical for production voice agents, which need to get account details, names, currency amounts and more right to be trusted by users, at the low latency that conversational experiences demand. Overview of Pronunciation Robustness Each model generates audio for 454 sentences containing 701 target words or phrases, across four categories: 1. Contextually appropriate (words read differently depending on context, e.g., a wound that is bandaged vs. a bandage that is wound) 2. Expanding shorthand (numbers, dates, units and notation read out naturally,…
Context
Artificial Analysis's new Pronunciation Robustness benchmark checks whether text-to-speech models say tricky words correctly, a question its existing tests leave open. Its TTS Arena measures which voices listeners prefer, and Word Error Rate checks whether the right words come out. Neither shows whether a model reads St. as Saint in St. Mary's and as Street in Church St., which matters for voice agents that read out account details, names and currency amounts.
Each model reads 454 sentences containing 701 target words or phrases in four categories: words whose pronunciation depends on , shorthand such as Chapter XVII or 1 tsp of sugar that should be read out naturally, exact sequences such as file paths, codes and email addresses that must be spoken exactly, and standalone names and terms such as Arkansas or façade. Sentences go in as written, with no normalization beyond each model's default setting. Screened human listeners then judge each target against pronunciations agreed in advance, and the score is the share of judgements marked correct.
At launch, Google DeepMind's Gemini 3.1 Flash TTS led at 88.1%, followed by SpaceXAI's TTS at 87.6%, ElevenLabs's Eleven v3 at 85.6% and v3 Conversational at 84.8%, and Alibaba's Qwen-Audio-3.0-TTS-Plus at 81.6%. The weak spots are the categories closest to the numbers and identifiers voice agents read out: expanding shorthand scored 62.4% and preserving exact sequences 62.9%, against about 86% for the other two. Artificial Analysis expects both scores to rise considerably when the input text is normalized first, especially for models that do not normalize by default, which suggests some of the gap can be closed by cleaning text before it reaches the model.
The ranking also differed from listener preference. Sonic 3.6 was first on Artificial Analysis's Provider Voice Arena at 1276 Elo but 11th on pronunciation at 74.5%, while Gemini 3.1 Flash TTS was ninth on the arena at 1201 Elo and first here. The voice people like best is not a reliable guide to the one that reads a confirmation code correctly.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…