Public STT benchmarks were not built for voice-agent speech and contain transcript errors that penalize correct models. This update adds a held-out agent-speech set and cleaned data, so model comparisons for voice agents are more trustworthy.
Key takeaways · AI-distilled
AA-AgentTalk carries 50% of the score (469 samples, about 250 minutes); cleaned VoxPopuli parliamentary speech and Earnings22 earnings calls get 25% each, about 8 hours of audio in total.
Artificial Analysis dropped the AMI-SDM meeting dataset because its transcript errors were too extensive to fix without many judgment calls, and built a normalizer on Whisper's English normalizer so formatting differences stop inflating WER.
ElevenLabs Scribe v2 leads overall at 2.3% AA-WER, ahead of Gemini 3 Pro (2.9%), Mistral Voxtral Small (3.0%), Gemini 2.5 Pro (3.1%) and Gemini 3 Flash (3.2%).
Leaders differ by dataset: on AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → speech Scribe v2 posts 1.6% ahead of AssemblyAI Universal-3 Pro (2.3%); NVIDIA Parakeet TDT 0.6B V3 places third on earnings calls (4.9%); Gemini 3 Pro leads parliamentary speech at 1.7%.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
