token, which is the basic building block of the actual TTS model. But, at this point, there's still discrepancy between the codec and the TTS model. The codec factorizes tokens into semantic and acoustic parts. But, the autoregressive prediction model expects indivisible audio tokens. Here's how this conflict gets resolved. After quantization, each audio frame gets one semantic token ID, which comes from the VQ codebook, and multiple acoustic IDs because FSQ gives you a separate integer for every dimension. From here, we'll treat audio tokens, or rather audio sub-tokens, similarly to text tokens. Remember, an LLM contains an embedding table that maps token IDs to internal continuous representations. All we need to do is extend this table with new entries for audio tokens. That is, about 8,000 more for the codebook tokens and groups of 20 buckets for each acoustic feature. Once semantic and acoustic tokens are looked up in the LLM embedding table, we can collapse their stacks by summing up the embeddings. And from here on, all tokens share the same interface and can therefore flow through a regular LLM. Well, almost regular because we'll have to change its output dictionary. Swap out the text vocabulary and swap in the semantic codebook. This way on each iteration, the LLM produces a semantic token ID. The acoustic part though needs special treatment. As we discussed earlier, acoustics don't have the same clustered nature. It's a lot harder to discretize and predict them using an inherently discrete LLM. If you've been following my channel, you already know about flow matching, the generative algorithm behind modern image generation. It learns a velocity field that transports noise directly towards the data distribution in a fixed number steps. The same idea applies to audio. Starting with a noisy vector and refining it eight times, the output becomes a clean prediction of the acoustic latent. The conditioning signal is the last hidden state of the LLM, which has been shaped by the voice reference from the context window. This is how the speaker's timbre and acoustic character get transferred because the LLM has been attending to the voice reference all along. By applying FSQ to the flow matching output, we get our stack of acoustic token IDs. In the next auto regressive step, all output token IDs make their way back to the input and the cycle repeats exactly 12.5 times per second. To produce an actual waveform, the output token IDs are dequantized into real values and passed through the decoder part of the codec. This is quite an involved journey for the clean acoustic latent. Initially, it's a continuous vector, then quantized, immediately dequantized, then expanded into another latent space by the codec decoder. A similar complexity is present in the loss function. The codec loss has five separate terms, including distillation from Whisper and an adversarial term. And the full Voux roll TTS adds cross entropy for semantic tokens, low matching loss on acoustics, and direct preference optimization for both. This goes to show that text to speech is a really difficult problem. It's an active field of research that hasn't yet converged to a simple and canonical solution. Today, most voice systems in production take the modular approach because it's customizable and interpretable. Now, the Voux roll speech detects and text to speech models are real-time in isolation, but the LLM in the middle is the limiting factor for a voice to voice system. The text bottleneck drops the audio and together with it, emotions like sarcasm or frustration that might actually be relevant in addressing the user query. In the next video, we'll look at models