Audio tokenization: one semantic token plus seven acoustic ones, with injected delays

to switch turns and can continue uninterrupted. So, how does Moshie work under the hood? Well, to generate audio, we need to split it into small building blocks that can be predicted one at a time. Ideally, we could predict raw amplitudes like my animation is showing. But, for real-time speech, that would require at least 24,000 sequential predictions per second, which is a challenge even with the newest GPUs. Instead, the most common approach is to segment the stream into fixed-size frames, something like 80 milliseconds. These are the audio equivalent of text tokens. Though, the analogy is not perfect because audio is a richer signal. It combines semantics, or what is being said, with acoustics, or how it's being said. Now, An and his team proposed this decomposition back at Google with Audio LM. Following the same principle, Moshi decomposes a single audio frame into one semantic token and seven acoustic ones. Now, these sub-tokens do indeed act very similarly to text ones. They can be mapped to integer token IDs, as well as real-valued embeddings. Audio tokenization is an entire subfield itself, and I covered it in this previous video. Even though my diagrams are showing a single audio stream for the assistant, there are actually eight of them, one for each sub-token. By keeping them separate, we can artificially inject delays between them. Specifically, acoustic tokens are delayed by two frames during pre-training and one frame during fine-tuning and inference. Why is that helpful? Because we're imposing a conceptual hierarchy here. First, generate coarse semantics, the equivalent of phonemes or words, and only then vocalize them with fine-grained acoustics. And the conceptual hierarchy doesn't