- Category
- AI Tools
- Rank
- No. 343Tools index
Previous survey · No. 338 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- karpathy
- GitHub
- 30.9k stars
- Date
About
Andrej Karpathy's LLM training implementation in pure C/CUDA — small, readable, performant reference for understanding GPT training from the metal up.
What it does
It trains and fine-tunes GPT-style language models through explicit CPU and GPU implementations. A parallel PyTorch version creates checkpoints and reference tensors, while compiled programs perform forward passes, backpropagation, optimization, validation, and text sampling.
Why it's ranked here
The project makes model training unusually inspectable without stopping at toy pseudocode. Its CPU implementation exposes the full algorithm, while the CUDA path adds mixed precision, distributed training, optimized libraries, and profiling. Reference comparisons keep the educational code tied to concrete numerical behavior.
What's good
The CPU implementation keeps tensor storage, attention, normalization, gradients, and AdamW visible in ordinary loops. Tests compare logits, loss, gradients, and repeated training steps against PyTorch outputs. The GPU build supports three precision modes, optional cuDNN attention, NCCL, MPI, and profiling.
Tradeoffs
Serious training requires substantial hardware and a more involved native toolchain. Multi-GPU work depends on CUDA, NCCL, and optionally MPI. cuDNN attention needs separate frontend headers and increases compile time. The simpler CPU and single-GPU floating-point variants are frozen reference snapshots rather than the main development path.
How to use it well
Use it to study training internals, validate custom kernels, reproduce GPT-2 experiments, or compare readable implementations with optimized GPU paths. Start with the CPU tutorial or frozen single-GPU path, then move toward mixed precision and distribution. It does not provide a hosted API or packaged application layer.
Technical notes+
README.md centers the workflow on GPT-2 reproduction and describes CPU, single-GPU, multi-GPU, and multi-node runs. Makefile builds train_gpt2.c, test_gpt2.c, train_gpt2.cu, test_gpt2.cu, and frozen FP32 CUDA variants; it detects OpenMP, NVCC, NCCL, MPI, and optional cuDNN. requirements.txt supplies the Python-side artifact and reference stack. train_gpt2.py writes checkpoints and debug tensors consumed by C, while test_gpt2.c checks logits, loss, gradients, and ten optimizer steps. doc/layernorm/layernorm.md derives LayerNorm from tensor math through pointer indexing. llmc/cudnn_att.cpp builds and caches cuDNN frontend graphs for causal attention and its backward pass. profile_gpt2cu.py drives Nsight Compute and summarizes kernel time, memory traffic, tensor utilization, and instruction counts.
Observed
- Primary languages
- C and CUDA, with Python reference and support scripts
- Build surface
- Make builds CPU and CUDA training, testing, and profiling executables
- Python dependencies
- PyTorch, NumPy, tiktoken, Transformers, Datasets, Requests, and tqdm
- Interfaces
- Local command-line executables plus shell and Python workflows
- Platform support
- CPU build logic covers Linux, macOS, and Windows; CUDA builds require NVCC
- Distributed training
- NCCL supports multiple GPUs, with MPI, shared filesystem, or TCP initialization for multiple nodes
- Precision modes
- FP32, FP16, and BF16 are selectable at build time
- Validation approach
- C tests compare logits, loss, gradients, and optimizer-step losses with PyTorch reference data
Read from README.md, Makefile, requirements.txt, doc/layernorm/layernorm.md, test_gpt2.c, train_gpt2.c, train_gpt2.py, train_llama3.py, profile_gpt2cu.py, llmc/cudnn_att.cpp.
What it can do
Train GPT language models
Text dataset and training parameters → Trained GPT model weights
Execute forward pass inference
Model weights and input text tokens → Generated text tokens
Perform gradient computation and backpropagation
Model weights, training data batch, and loss function → Computed gradients for model parameters
Optimize model parameters
Computed gradients and learning rate → Updated model weights
Accelerate training with CUDA
Training data and model on GPU → Faster training execution on GPU hardware
Demonstrate transformer architecture implementation
Source code examination → Understanding of GPT training mechanics
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.