Clip / Education

How BPE tokenizers handle code, and what happens to unusual identifiers

From Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs · ≈21:37

“we're going to be using like a character-level tokenizer because it's going to be easy for this project, but most big labs, they don't use character-level tokenization.”
AI Engineer

“Uh and the way it works, it will just look at common patterns. Uh so, if you have like a lot of training data that is that is code itself, it will it will see, and then it will realize, "Okay, for loops seem to be like a good candidate for for that to be a token."”
AI Engineer

“So, it will look at all these different tokens, and then create this tokenizer based on the common uh relationship between them.”
AI Engineer

“So again like there's no there's no specific way there's no like human in the loop in this process. It depends on your training data that you use to train this tokenizer.”
AI Engineer

“Like if your your if your variable names are like very strange like random characters and yeah probably that's not going to be in the tokenizer.”
AI Engineer

What’s in it

Answers a real question about tokenizing programming languages — common keywords and identifiers become tokens from corpus statistics, and unusual names fall back to character level.

Watch this moment

Clip transcript

So, when it comes to tokenization, how does that work? So, yeah, as I mentioned to you before, uh we're going to be using like a character-level tokenizer because it's going to be easy for this project, but most big labs, they don't use character-level tokenization. They They use what I was mentioning earlier, the BPE tokenizer or like byte push Essentially, they they what they do is that they You You look at your training data. Let's say you have like this many trillions of tokens, and your training data is going to be including like code, as you mentioned yourself. Uh and the way it works, it will just look at common patterns. Uh so, if you have like a lot of training data that is that is code itself, it will it will see, and then it will realize, "Okay, for loops seem to be like a good candidate for for that to be a token." So, the for is going to be for sure a token. And then you might have some like enumerate. Enumerate you Do you see quite commonly in the um in the training data, so that will be another token. So, it will look at all these different tokens, and then create this tokenizer based on the common uh relationship between them. Uh of course, like maybe some languages that are like maybe Pascal that might not be very common in the language, so this the keywords of Pascal might not be like very common. Although, I do think that there probably is like good representation in the tokenizers, too. But some other of those like crazy like white space-only like languages that this these ones probably are not going to be work very well. My question was more on the So, one thing is the keywords of the programming language, but then you have your own variable names. Which are not common among different programs. >> I I see that's what you mean. Making them as tokens, I'm not sure how does that help. So again like there's no there's no specific way there's no like human in the loop in this process. It depends on your training data that you use to train this tokenizer. Uh if your code like there's like foo and bar like that's like common variables these were probably are going to be tokens. Like if your your if your variable names are like very strange like random characters and yeah probably that's not going to be in the tokenizer. And when this happens uh the tokenizer will fall back to character level or