- Category
- Developer Tools
- Pricing
- Open Source
- Type
- TOOL
- Builder
- openai
- GitHub
- 18.8k stars
- Added
- May 22, 2026
About
OpenAI's fast BPE tokenizer — count tokens, encode/decode for GPT models, written in Rust with Python bindings.
Why it made the leaderboard
OpenAI's fast BPE tokenizer, written in Rust with Python bindings — count tokens and encode/decode for GPT models exactly instead of estimating context budgets.
Tags
tokenizerbpeopenaillmpython
Tech Stack
PythonRust
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.