- Category
- Education
- Rank
- No. 1131Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- karpathy
- GitHub
- 4.3k stars
- Date
About
Andrej Karpathy's autoregressive character-level language model for generating more things — names, words, anything — from a small training set.
What it does
Give it a line-separated collection of examples, then train a model that predicts each next character. It can use approaches ranging from simple character-pair counts to recurrent networks and a Transformer. During training, it saves progress and models, while printing newly synthesized items that resemble the input collection.
Why it's ranked here
This is a compact teaching tool with unusually broad model coverage for one script. It lets learners compare basic counts, multilayer perceptrons, recurrent designs, and Transformer-style attention on the same small problem. Its narrow scope keeps the mechanics visible, but also makes it a poor fit for production language-model infrastructure.
What's good
The common input format makes experiments easy to understand: one example per line. PyTorch is the only stated requirement, and training can run without special hardware. The script saves logs and model state, prints samples during training, and can separately load the best saved model for interactive sampling.
Tradeoffs
The project intentionally avoids a heavyweight library design, extensive switches, pretrained GPT-2 weights, dropout, and weight-decay machinery. The CNN is described as unfinished, while the source says LSTM was not implemented. Configuration requires reading the argument parser and code, so discoverability and abstraction are deliberately limited.
How to use it well
Use it for learning, demonstrations, and quick experiments with short categorical strings such as names or word lists. Start with the included example data, compare model families, then tune the compact implementation directly. Do not choose it when you need pretrained models, a polished reusable library, or production serving infrastructure.
Technical notes+
README.md documents a CLI invocation of python makemore.py -i names.txt -o names and a --sample-only mode that reloads the best model. makemore.py is a single PyTorch script using argparse, Dataset, DataLoader, and SummaryWriter. The supplied excerpt defines ModelConfig, explicit masked multi-head attention, Transformer blocks, a causal bag-of-words model, and custom RNN and GRU cells. Its module docstring records deliberate removals from minGPT: GPT-2 pretrained initialization, dropout, weight decay, and related parameter-group complexity. names.txt supplies the line-oriented example corpus, and LICENSE contains the MIT terms.
Observed
- License
- MIT License
- Primary language
- Python
- Install surface
- No package installation is described; the README states PyTorch is the only requirement.
- Interface
- Command-line script for training and sample-only generation.
- Input format
- One text file with one training item per line.
- Project structure
- The described implementation is concentrated in one hackable script.
Read from README.md, makemore.py, LICENSE, names.txt.
What it can do
Generate new names from training examples
Dataset of example names → New generated names
Generate new words from training examples
Dataset of example words → New generated words
Train character-level language model
Text training dataset → Trained autoregressive model
Generate text sequences character by character
Trained model and optional seed text → Generated character sequences
Learn patterns from small text datasets
Small collection of text examples → Model weights capturing text patterns
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.