Clip / Education

Vocabulary size can dominate a small model's parameter count

From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈14:38

Concrete arithmetic showing why a character-level tokenizer is the right call at 384-dimensional embeddings — the embedding table would otherwise outweigh the entire network.

What’s in it

Concrete arithmetic showing why a character-level tokenizer is the right call at 384-dimensional embeddings — the embedding table would otherwise outweigh the entire network.

Watch this moment ↗

Clip transcript

such a big tokenizer, that would also be like more than twice the size of our models cuz if you just multiply this, our uh embedding size is 384 for this model that we're going to be training. So, that's already like 25,000 parameters and in that case like if we used GP the tools vocab which is 50,000, that would be 19 million parameters which is like more than three times the model which wouldn't make sense in our case.

Intel