kilohertz. In 2016, DeepMind's WaveNet was the first neural network that attempted to model a raw waveform directly. Given that this was about a decade ago, its outputs were remarkably natural-sounding. Here's one example. >> The Blue Lagoon is a 1980 American romance and adventure film directed by Randal Kleiser. >> However, WaveNet was notoriously slow since it had to generate 24,000 samples per second sequentially. For real-time, a new sample would have to be generated every 42 microseconds. Nvidia actually pulled it off with a V100 GPU by fusing the whole WaveNet model into a single kernel and caching the weights in the registers. But that's already scraping the bottom of the barrel in terms of optimization headroom and we We that model sizes scale a lot faster than hardware capabilities. A more pragmatic approach came 5 years later with models like SoundStream from Google and EnCodec from Meta. Instead of predicting individual samples, they worked with frames. These were sequences of 240 samples or 10 milliseconds of audio. And this strategy survived until today. Now, we've already encountered audio frames in the context of speech-to-text, where the audio stream was split into fixed-size windows and each window was converted into a continuous embedding. In contrast, in the TTS world, audio frames are treated as indivisible units, much like text tokens. Predicting a single audio token ID instead of all the values in the frame makes the problem more tractable. For instance, VALL-E TTS uses 80-millisecond audio frames. This brings the real-time requirement down from 24,000 to only 12.5 sequential predictions per second. But this brings an interesting challenge, audio tokenization. We know, of course, that text tokenization relies on a vocabulary that maps integer indices to character strings. What exactly is the audio equivalent of character strings here? In theory, we could simply store raw sequences of amplitudes, but their space is combinatorially huge. Plus, raw amplitude captures the wrong level of abstraction. The same word spoken at a slightly different volume or speed becomes an almost entirely different vector. >> What's the meaning of life? >> What's the meaning of life? >> What's the meaning of life? >> Instead, audio tokens are compressed in a latent space, where vector similarity is meaningful. Because these entries are called codes, the vocabulary is actually called a codebook. So, how is this codebook built? Well, first of all, we need a way to induce this latent space. The standard machine learning solution across all modalities is an encoder-decoder model. It takes in the raw signal, which is highly dimensional, passes it through a low-dimensional bottleneck, and then reconstructs it. In the audio world, this model is known as a codec, which it comes from coder-decoder. It plays a similar role to the VAE model used for image generation, though it's trained with a very different loss. Modern codecs apply vector quantization or VQ right after the bottleneck. They define a codebook of the desired size. In VQ Voxel's case, a bit over 8,000 codes with 256 dimensions each. The compressed latent is then snapped to the nearest entry in the codebook. This gives us a code, which is a vector of 256 values, as well as an integer index. The code is then passed on to the decoder to complete the forward pass. This way, the codebook entries are learned together with a codec. The integer index will only serve later when training the actual text-to-speech model. Now, this is a valid tokenization scheme, but arguably, audio is too rich of a signal to be captured in around 8,000 tokens. One option would be to simply increase the codebook size. After all, a text vocabulary can go up to 250,000 tokens. However, empirically, large audio codebooks lead to a phenomenon called codebook collapse, where most entries remain unused. There are two effective alternatives to vector quantization. One is residual vector quantization or RVQ. After the first level of quantization, there's inevitably a small error or residual. That's the difference between the original embedding and the code we picked. Instead of ignoring it, RVQ quantizes the residual with a second codebook, then a third, and so on. This way, each layer of quantization captures finer and finer detail. The second alternative to vector quantization makes a different trade-off. Finite scalar quantization, or FSQ, treats vector dimensions independently. For any particular feature, its range of values is split into n buckets, say 20. For simplicity, let's assume all features fall between -1 and +1. Quantizing an input feature means snapping it to the middle of its bucket. So, for instance, -0.72 turns into -0.75. FSQ is more memory-efficient since there's no codebook needed, and you can trivially tweak its precision by increasing the number of buckets. However, it fails to capture correlations between dimensions. Plus, every single feature in the original latent now gets its own integer index. This is different from vector quantization where the entire input embedding is mapped to a single integer. Since these three types of quantization make different trade-offs, modern audio models mix and match them in various ways without having converged to a standard solution. Let's see how VALL-E tackles this challenge. The quantization strategy used by the VALL-E codec reflects an older philosophy about speech. The philosophy comes from AudioLM, a Google model from 2022 that decomposes audio into two orthogonal aspects: what is being said, and how it's being said. Every audio frame is decomposed into two embeddings. The semantic part captures phonemes or words and their meaning. And the acoustic part reflects aspects like the speaker timbre, prosody, emotion, microphone characteristics, breath sounds, and so on. The VALL-E codec quantizes the two latents separately. The semantic one with vector quantization and the acoustic one with finite scalar quantization. This choice is quite intuitive. Semantics are tied to words, which are naturally clustered by meaning and can more credibly be organized into a codebook. Intuitively, similar words should map to the same code. Acoustics, on the other hand, are more continuous in nature and require higher precision, which is what FSQ offers.