Intel / article

How Together AI built the world’s fastest speech-to-text stack

Source: together.ai
Author: Together AI
Date: Published May 29, 2026

Context

Speech recognition can spend more time coordinating computation than performing it. In Together AI's deployment of NVIDIA's Parakeet model, the decoder repeatedly asked the CPU whether a step produced text. Each decision interrupted GPU execution, even though the underlying calculations took only microseconds. The engineering account identifies that round trip as a bottleneck.

The team moved the conditional decision onto the GPU and captured the decoder loop as one executable graph. It reports a two- to threefold faster decoder. That improvement concerns one component of transcription; the same account separately addresses audio preprocessing, data transfer and delays. Faster arithmetic alone would have left the coordination overhead intact.

Terms in this piece · Glossary

streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Key quotes

“The faster of the two, NVIDIA Parakeet-TDT 0.6B v3, can transcribe roughly 20 hours of speech, about the runtime of the Harry Potter film franchise, in under 10 seconds.”
Together AI

“The same Harry Potter corpus as audiobooks is 5 to 10 GB, roughly three orders of magnitude larger than the text.”
Together AI

“The CPU leaves the decoder’s inner loop, and the result is a 2 to 3x faster decoder.”
Together AI

“Under load for streaming workflows, p50 and p90 latency looked healthy, but p95 would periodically spike by about 200 ms.”
Together AI

“The lesson was to keep profiling beyond the model. GPU time, queue depth, and model execution all looked normal; the latency spike lived in the Python runtime.”
Together AI

Read the source www.together.ai