- Category
- AI Tools
- Rank
- No. 877Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- karpathy
- GitHub
- 518 stars
- Latest release
- v0.1.0
- Date
About
Fast BPE tokenizer training in Rust by Andrej Karpathy. The missing tiktoken training code.
What it does
rustbpe learns a byte-level vocabulary from text by repeatedly combining frequent adjacent token pairs. It splits input with a GPT-4-style pattern by default, processes buffered Python iterators in Rust, and can encode, decode, or hand learned ranks to tiktoken.
Why it's ranked here
The narrow scope is persuasive: training, compatibility export, and little Python-side setup. Parallel pair counting and batch encoding address expensive work, while reference comparisons and cross-platform automation provide useful confidence. The repository also openly acknowledges uncertainty about the Rust implementation’s structure.
What's good
It accepts an iterator instead of requiring one giant input string, buffers ingestion, and releases Python’s interpreter lock for parallel Rust processing. Repeated text chunks are collapsed into weighted unique sequences. Custom splitting patterns are supported, and learned byte ranks transfer directly into tiktoken for inference.
Tradeoffs
This is deliberately narrower than a general tokenizer framework. The documented export leaves special tokens empty, and the public workflow centers on GPT-style byte pair encoding rather than multiple tokenizer families. Training requires at least 256 vocabulary entries. The author also flags possible weaknesses in Rust arrangement or implementation.
How to use it well
Use it when Python owns the data pipeline but tokenizer training needs a compact Rust engine. Stream corpus lines into training, validate the learned vocabulary against representative text, then construct a tiktoken encoding for serving. Choose another toolkit when you need accumulated tokenizer variants or documented special-token training.
Technical notes+
Cargo.toml builds both cdylib and rlib targets, uses PyO3 for bindings, Rayon for parallelism, fancy-regex for splitting, and an octonary heap for merge selection. In src/lib.rs, count_pairs_parallel aggregates pair frequencies and affected word positions across Rayon workers; train_core_incremental updates counts around merged pairs, and stale heap entries are refreshed and requeued when their stored counts differ. pyproject.toml uses maturin, declares Python 3.9 through 3.13, and has no runtime Python dependencies. tests/python/test_tokenizer.py compares implementations and tiktoken export. .github/workflows/ci.yml runs Rust and Python tests on Ubuntu and macOS, plus formatting and Clippy checks. .github/workflows/release.yml builds Linux, Windows, and macOS wheels plus a source distribution.
Observed
- License
- MIT
- Primary language
- Rust, with Python bindings and Python tests
- Install surface
- PyPI package installable with pip; source builds use maturin
- Interfaces
- Python library and Rust library targets; no CLI documented
- Python support
- CPython 3.9 through 3.13 declared
- Release platforms
- Linux x86_64 and aarch64, Windows x64, macOS x86_64 and aarch64
- Testing structure
- Rust tests and a dedicated Python test directory are documented
Read from README.md, Cargo.toml, pyproject.toml, src/lib.rs, tests/python/test_tokenizer.py, LICENSE, .python-version, .github/workflows/ci.yml, .github/workflows/release.yml.
What it can do
Train BPE tokenizer models
Text corpus/dataset → Trained BPE tokenizer model
Generate BPE token vocabulary
Training text data and vocabulary size parameters → BPE vocabulary file
Tokenize text using trained BPE model
Raw text and trained BPE model → Sequence of BPE tokens
Create tiktoken-compatible tokenizer
Training corpus and tokenizer configuration → tiktoken-compatible tokenizer model
Encode text to token IDs
Text string and BPE model → Array of token IDs
Decode token IDs to text
Array of token IDs and BPE model → Reconstructed text string
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.