So, what exactly enables this performance? Well, let's work it out backwards from the output. Here, I've modified Vaux's official Hugging Face space to surface the raw output tokens. If you look at their timestamps, you'll notice that a new output token is emitted every 80 milliseconds. This is the cadence of a streaming model. You'll also notice two special tokens, padding, which communicates "I'm still listening" or "I'm still processing", and the word boundary placed right before a stream of actual text tokens. These are all artifacts of the recent modeling framework called Delayed Streams Modeling or DSM. It comes from a Paris-based company called Cute AI. We'll hear more from them in the next video. DSM introduced this idea of a heartbeat. Both audio and text streams advance at a regular cadence, say 80 milliseconds. So, in each heartbeat, a speech-to-text model processes an audio frame of 80 milliseconds and also outputs a text token corresponding to a previous audio frame. The text stream is therefore delayed by a few heartbeats, since obviously you can only transcribe things that have already been said. That's why Vox Troll's output starts with a bunch of padding tokens. Say we set the desired delay to 160 milliseconds. Then the first two pads enforce this delay, and the rest are just accounting for silence, since I didn't start speaking immediately after pressing record. If you're a machine learning practitioner, your first question might be, "How do you even train such a thing in the first place?" Well, the first requirement is training data that aligns audio and text at a word level, so that the model learns precisely what to say and when. You'll need something like this JSON file, where every word comes with its own start and end timestamps. And what about the configurable delay between audio and text? Say that the model supports delays between 80 to 2400 milliseconds, that is from 1 to 30 heartbeats. To support this entire range, that DSM framework picks a random delay for each training batch, offsets the text stream by that amount, and fills the gap with padding tokens. Delayed streams modeling is a huge enabler for streaming models, but if we're being realistic, it's not rocket science. It's quite an obvious algorithm. So, why did it take so long for it to be published in 2024? Well, the truth is streaming models have existed for a long time, but they used to be implemented with recurrent neural networks, which are a very natural architectural fit. The original transformer, however, was less so. So, the true value of DSM is to bring streaming to the transformer in a way that enables pre-trained LLMs to bootstrap speech models. Contrary to common belief, the original transformer as published in 2017 was intended for offline machine translation, mapping a piece of text from a source to a target language. The language modeling revolution actually happened later. I covered this in one of my previous videos. Now, speech-to-text is inherently a translation problem, not between languages but between modalities. That's why offline speech-to-text models like OpenAI's Whisper were based on the original transformer architecture. At the time, it had two core components, an encoder to process the source sentence, and a decoder to generate the translation auto regressively. Because the source sentence was assumed to be available in full, the encoder used bidirectional attention, where each token embedding depended both on its left and right context. This is what limited Whisper's real-time capabilities. It needed a full 30-second context window before it could start transcribing. Of course, people have hacked around this limitation. Take Whisper streaming, for example. It wraps the existing offline model with an API that mimics real-time behavior by keeping an internal audio buffer capped at 30 seconds. Once the audio buffer is filled, Whisper streaming makes its first call to Whisper. When a new chunk arrives, it displaces the oldest one in the queue and triggers yet another call to Whisper. This did simulate real-time behavior with just a few hundred milliseconds delay and only a small hit in accuracy. But it was ultimately still a wasteful hack since the same audio chunks were transcribed repeatedly. Plus, there was a mismatch between training and inference. Whisper was trained on full sentences, but Whisper streaming was calling it with incomplete sentences and even incomplete words. In contrast, modern speech-to-text