nanochat

Why it's ranked here

The project offers an unusually legible route through the complete model-building process while retaining serious performance work. Its depth-based configuration makes controlled scaling experiments approachable, and the reference training workflow provides a repeatable target. The strongest results still assume expensive data-center GPUs, so its simplicity does not remove the infrastructure requirement.

What's good

One complexity control coordinates width, attention heads, learning rates, weight decay, and training horizons. The code supports single-GPU execution through automatic gradient accumulation and distributed execution with sharded optimizer state. Evaluation covers multiple-choice knowledge, grade-school mathematics, conversational data, and a broad core metric. Explicit precision handling makes hardware behavior easier to inspect.

Tradeoffs

The reference workflow targets eight H100 GPUs, while one GPU takes roughly eight times longer. GPUs below 80GB may require batch-size tuning to avoid memory exhaustion. CPU and Apple Silicon examples intentionally train much smaller models with weak results. Less-tested PyTorch backends may have sharp edges, and reinforcement learning lacks gradient scaling for float16 training.

How to use it well

Use nanochat for learning, reproducible model-training experiments, scaling-law studies, and testing principled architecture or optimizer changes across several model sizes. Start with a smaller depth for fast iteration, track validation loss, capability metrics, memory use, and throughput, then scale promising changes. It does not replace access to large GPU capacity when strong results matter.

Observed

Primary language

Python

Runtime requirement

Python 3.10 or newer

Packaging and installation

uv project with mutually exclusive CPU and GPU extras

Interfaces

Training scripts plus an interactive chat CLI

Platform support

CUDA, CPU, and Apple Silicon MPS paths are provided

Core framework

PyTorch 2.9.1