Intel / article

ttok 1.0

Source: simonwillison.net
Date: Published Oct 9, 2026

Context

Simon Willison released ttok 1.0 on October 9, 2026, according to his post. ttok counts and truncates text by , the chunks of text a model reads, which are what usage and limits are measured in. After releasing version 0.4 he noticed it still defaulted to the GPT-4 tokenizer when it should default to the GPT-5/GPT-6 one, and he treated switching the default as a reason to ship 1.0.

The change assumes GPT-6 tokenizes text like the GPT-5 family, and Willison says OpenAI has not confirmed that. His support is a commit by William Liu reporting that seven GPT models (5.5, three 5.6 variants and three GPT-6 variants) each counted 44,794 tokens on a 31-fixture test corpus and agreed on every fixture. That makes a match likely for that corpus, and it is an outside experiment rather than an OpenAI statement.

Key takeaways · AI-distilled

Simon Willison shipped ttok 1.0 after noticing that the just-released 0.4 still counted and truncated with the GPT-4 tokenizer by default; the 1.0 release switches the default to the GPT-5/GPT-6 tokenizer.
OpenAI has not confirmed that GPT-6 shares the GPT-5 family's tokenizer, and Willison notes an open, contested issue about it, so the new default rests on outside testing rather than an official spec.
The evidence Willison cites is an experiment by William Liu: seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) each reported 44,794 tokens and agreed on all 31 test fixtures, suggesting GPT-6 brings no input-count change on that corpus.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source simonwillison.net