Intel / article

How speech models fail where it matters the most and what to do about it

Source: together.ai
Author: Together AI
Date: Published Feb 23, 2026

Context

A speech recognizer can perform well on ordinary sentences yet miss the street name that determines a destination. Together AI's SF Streets tested 2,262 utterances from 78 participants across 15 speech models and variants. The models averaged a 39% street-name transcription error rate. Short names provide little surrounding to help resolve unfamiliar pronunciations. The study report explains this gap.

Reported accuracy was 46% for speakers whose primary language was not English, versus 64% for English-primary speakers: an 18-percentage-point difference. This makes speaker diversity and destination-name accuracy consequential evaluation targets for navigation systems, even when a model's aggregate speech score looks strong.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“We demonstrate that voice recognition systems struggle to understand street name pronunciations when speakers have diverse linguistic backgrounds — with an average transcription error rate of 39% across 15 state-of-the-art models, and an 18% accuracy gap between non-English and English primary speakers.”
Together AI

“For example, Whisper-Large achieves a respectable general Word Error Rate of 14%, but its specific error rate on street names rises to 27%.”
Together AI

“This technical failure translates directly into a more practical operational friction: to measure the real-world consequences, we mapped the transcribed street names to geographic coordinates using the Google Maps API; we found that mis-transcriptions for non-English primary speakers resulted in routing destinations that were, on average, 2.40 miles away from the intended location.”
Together AI

“We prompted the open-source XTTS model to generate speech in a foreign language like Spanish, but we injected specific English street names into the prompt. This forced the model to apply non-English phonetic rules to English words.”
Together AI

“Fine-tuning Whisper-Base on this small synthetic set yielded a 60% relative improvement in accuracy from the base model, with the biggest improvements happening among non-English primary speakers.”
Together AI

Read the source www.together.ai