tiktoken

Why it's ranked here

The case rests on focused engineering: Rust handles performance-sensitive work, Python exposes practical batch and array interfaces, and the registry supports custom encodings. The published speed claim is strong, though its comparison uses older pinned packages.

What's good

Encoding is lossless at the byte level and accepts arbitrary text. Batch work uses Python threads while Rust releases the interpreter lock. Array output avoids copying tokens into a Python list. Special-token handling defaults to rejecting accidental reserved text.

Tradeoffs

String decoding replaces invalid UTF-8 by default, so exact recovery requires byte decoding or strict error handling. Model-prefix matching can accept nonexistent model names. Encoding data may require network access and local caching. One completion-oriented interface is explicitly unstable.

How to use it well

Use it in Python pipelines that count, inspect, batch, or transform tokens for known OpenAI model families. Choose encodings explicitly when model mapping fails or ambiguity matters. It supports custom vocabularies, but it is not a model client or inference service.

Observed

Primary implementation

Rust core with Python bindings and Python library wrappers

Install surface

Published as a Python package installable with pip from PyPI

Python requirement

Python 3.9 or newer

Packaging

Setuptools builds a PyO3 Rust extension through setuptools-rust

Interfaces

Python library API for encoding, decoding, batching, model mapping, and NumPy output

Extension mechanism

Namespace-package plugins can register additional encoding constructors

Wheel targets

Build configuration covers Linux and macOS, skips 32-bit Linux and Windows wheels, and targets Intel and Arm macOS

License declaration

Package metadata points to the repository LICENSE file