← All IntelClip / Education
Audio tokenizers are trained on mel spectrograms and are dataset-specific
From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈1:18:52
Explains why audio tokenization can't be hand-designed and why covering both speech and music in one tokenizer is the genuinely hard case.
What’s in it
- Explains why audio tokenization can't be hand-designed and why covering both speech and music in one tokenizer is the genuinely hard case.
Clip transcript
how long how long it takes to play. It's very hard. And it's not really something that you do by by just sitting down and thinking, "Okay, I'm going to tokenize this word to that." This This is not something that you like sit down and like decide. You use some kind of process in the you train like an audio tokenizer through You have a It's a very similar like case, I guess, how you train like a text tokenizer. You'd use your training data and you'd find like common patterns in the audio and then tokenize those patterns. Of course, like when we say audio, we don't don't always just mean like the actual like sample rate and like the the audio waves. Usually convert them to something that's a bit more easy to to tokenize, to to do this processing. The most common one is like is mel spectrograms. It's First, you convert your audio to mel spectrograms and then use this this this this essentially arrays of like of numbers to train your your tokenizers. And it will be very dependent on your training set. Like if you wanted to tokenize music and use a music data set, your audio tokens for music are going to be very different than if you had a voice data set. That's going to be like focusing on like human voice. Now, the hard part is what if you want to do both voice and music? That's where like That's That's very hard.
Recommended reads
- clipAudio tokenization: one semantic token plus seven acoustic ones, with injected delaysJulia Turc
articleOne Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30Cohere- clipWhy voice AI can't track time: tokens have implicit time but models don't use itJulia Turc
Comments
Checking sign-in…
Loading comments…