Vibeleaderboard
← Glossary
Glossary · models

token

Also: tokens

The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Models don't see letters or words; text is split into tokens ("understanding" might be "under" + "standing"). A rule of thumb: 1,000 tokens is about 750 English words. Code tokenizes less efficiently than prose, which is one reason large codebases strain models.

Tokens are the meter that matters everywhere: context windows are measured in them, API pricing is per million of them, and generation speed is tokens per second.

In the index