nanoGPT
github.com/karpathy/nanogpt- Category
- AI Tools
- Rank
- No. 134Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- karpathy
- GitHub
- 62.9k stars
- Date
About
A minimal, fast implementation for training and fine-tuning medium-sized GPT models from scratch. Features clean, readable code with just ~300 lines each for the training loop and model definition, making it easy to customize and experiment with.
What it does
nanoGPT is a compact Python workspace for building causal language models, preparing text datasets, training them, resuming checkpoints, and generating text. It can start with random weights or import GPT-2 weights, then run on a laptop, one GPU, or distributed GPU nodes.
Why it's ranked here
The appeal is transparency: the model, optimizer path, data loading, checkpointing, evaluation, and generation remain close enough to inspect and change. It also demonstrates serious distributed training. However, the repository explicitly calls itself old and deprecated, and directs most prospective users toward its newer cousin.
What's good
The included workflows span a tiny character model, GPT-2 fine-tuning, checkpoint evaluation, and distributed pretraining. Configuration overrides keep experiments direct. Training includes gradient accumulation, mixed precision, compilation, cosine learning-rate decay, validation checkpoints, optional experiment logging, and model-efficiency estimates without hiding those mechanics behind a framework.
Tradeoffs
Large-scale reproduction demands substantial hardware, time, dataset storage, and a fast cluster interconnect. CPU examples intentionally shrink context, batches, model depth, and training duration, producing worse samples. Configuration files execute as Python, which is flexible but deliberately described as a rough approach. The project is also explicitly deprecated.
How to use it well
Use it when learning or modifying the mechanics of GPT training matters more than framework abstraction. Start with the small Shakespeare workflow, then fine-tune an imported checkpoint or profile a changed model before scaling out. It does not document a hosted API or production serving workflow, so pair it with separate deployment infrastructure.
Technical notes+
train.py reads data/<dataset>/train.bin and val.bin through NumPy memmaps, supports scratch, resume, and GPT-2 initialization, and combines DDP, gradient accumulation, autocast, GradScaler, torch.compile, validation, checkpointing, and optional Weights & Biases logging. model.py defines GPTConfig and GPT with learned token and position embeddings, causal self-attention, a PyTorch scaled-attention fast path with a manual fallback, tied embedding and output weights, AdamW configuration, checkpoint import through Transformers, and autoregressive generation. configurator.py executes a supplied config file and applies type-checked --key=value overrides. sample.py loads local checkpoints or GPT-2 variants and samples with temperature and top-k controls. bench.py measures iteration time and model-flops utilization or emits profiler traces. data/openwebtext/prepare.py tokenizes OpenWebText into uint16 binary streams, while data/shakespeare/prepare.py creates smaller GPT-2-tokenized training and validation streams.
Observed
- Primary language
- Python, using PyTorch for model definition, training, profiling, and generation.
- Install surface
- The README instructs users to install torch, numpy, transformers, datasets, tiktoken, wandb, and tqdm directly with pip.
- Interface
- Script-driven command line workflows accept Python configuration files and typed key-value overrides.
- Platform support
- Documented execution targets include CPU, CUDA GPUs, Apple Silicon through MPS, and multi-node distributed GPU training.
- Checkpoint support
- Training can start from scratch, resume a saved checkpoint, or import GPT-2, medium, large, and XL weights through Transformers.
- Training data format
- Prepared training and validation token streams are stored as raw uint16 binary files and read with NumPy memmaps.
Read from README.md, bench.py, model.py, train.py, sample.py, configurator.py, config/eval_gpt2.py, config/train_gpt2.py, config/eval_gpt2_xl.py, config/eval_gpt2_large.py, config/eval_gpt2_medium.py, config/finetune_shakespeare.py, config/train_shakespeare_char.py, data/openwebtext/prepare.py, data/shakespeare/prepare.py.
What it can do
Train GPT models from scratch
Training dataset and hyperparameters → Trained GPT model weights
Fine-tune existing GPT models
Pre-trained model and fine-tuning dataset → Fine-tuned GPT model
Reproduce GPT-2 124M model
OpenWebText dataset → GPT-2 124M equivalent model
Execute model training loop
Model configuration and training data → Model checkpoints and training metrics
Define and initialize GPT model architecture
Model parameters and configuration → GPT model instance
Intel on nanoGPT
Tags
Featured in Intel
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.