← All IntelClip / EducationAudio tokenizers are trained on mel spectrograms and are dataset-specific
From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈1:18:52
Explains why audio tokenization can't be hand-designed and why covering both speech and music in one tokenizer is the genuinely hard case.
What’s in it
- Explains why audio tokenization can't be hand-designed and why covering both speech and music in one tokenizer is the genuinely hard case.
Clip transcript
how long how long it takes to play. It's very hard. And it's not really something that you do by by just sitting down and thinking, "Okay, I'm going to tokenize this word to that." This This is not something that you like sit down and like decide. You use some kind of process in the you train like an audio tokenizer through You have a It's a very similar like case, I guess, how you train like a text tokenizer. You'd use your training data and you'd find like common patterns in the audio and then tokenize those patterns. Of course, like when we say audio, we don't don't always just mean like the actual like sample rate and like the the audio waves. Usually convert them to something that's a bit more easy to to tokenize, to to do this processing. The most common one is like is mel spectrograms. It's First, you convert your audio to mel spectrograms and then use this this this this essentially arrays of like of numbers to train your your tokenizers. And it will be very dependent on your training set. Like if you wanted to tokenize music and use a music data set, your audio tokens for music are going to be very different than if you had a voice data set. That's going to be like focusing on like human voice. Now, the hard part is what if you want to do both voice and music? That's where like That's That's very hard.
Comments
Sign in to comment.
Loading comments…