← All IntelClip / EducationTokenizer size has to match your data budget
From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈4:36
“If you want to train, for example, a very big model that can that wants you want to generate text for like multiple languages, you'll need a huge tokenizer, which means you're going to need a huge amount of data as well to train it.”
“for a smaller model, a smaller tokenizer with with with smaller embeddings is what would work best and train the fastest when your when your data limited”
“Like if you check the difference between like GPT-4, GPT-4o, and GPT-5 or even even if you go before that, what you'll see mostly that usually the pre-training is very similar. It's the fine-tuning and post-training and essentially what you use with a with the same base model or like very similar base model.”
What’s in it
- A practical sizing rule from a working research engineer: a large multilingual vocabulary demands a correspondingly large corpus just to train the tokenizer.
Clip transcript
And we're going to go over over those in in a bit. The four building blocks you need to to train a model is First one is going to be the tokenizer. Uh And depending on what your use case is, you'd want to use a different tokenizer for that specific use case. If you want to train, for example, a very big model that can that wants you want to generate text for like multiple languages, you'll need a huge tokenizer, which means you're going to need a huge amount of data as well to train it. But for a for a smaller model, a smaller tokenizer with with with smaller embeddings is what would work best and train the fastest when your when your data limited, which is what we are right now. Next will be a model architecture. To be honest, like at most models like have at least at at the at the period that we're going to be working in were very similar. Like in in they were just decoder only causal decoder only models that had a very similar uh way of of using causal self-attention and the same like MLP layers and the same like layer norms and all kind of stuff, which we'll we'll get we'll get to it. Um So, if you know how to do this this like small models, it's very easy to go and do the same process for like bigger like newer models, but of course the newer models are going to be way more specialized for longer context and essentially essentially being able to scale this train the they're architected in a way to be able to scale training to as many tokens as possible, which you won't need in this case. And lastly it's the it's the training loop, which this is generally the most important part when you're training a new model. Like if you check the difference between like GPT-4, GPT-4o, and GPT-5 or even even if you go before that, what you'll see mostly that usually the pre-training is very similar. It's the fine-tuning and post-training and essentially what you use with a with the same base model or like very similar base model. And the way you train it actually makes a big difference in performances. And now we see for example uh Gemini 3 comes out and then has this this this many this good benchmarks and then 3.1 comes out that has like double the performance in some benchmarks, which is crazy. Like obviously it's very sim- the models are very similar, but actually during training the train the the train the new model in smarter ways to improve performance like very substantially. Now as for the end and lastly of course
Comments
Checking sign-in…
Loading comments…