MicroGPT-C
github.com/vixhal-baraiya/microgpt-c- Category
- Developer Tools
- Rank
- No. 2827Tools index
Previous survey · No. 2752 ·
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Models: Train & Run
- Interfaces
- SDK
- Builder
- @TheVixhal
- GitHub
- 853 stars
- Date
About
A minimal GPT implementation written in pure C with zero dependencies. Trains and runs a small transformer model from scratch, optimized for performance with AVX2 instructions.
What it does
A tiny character-level transformer that you train and sample from in one C program with nothing but the standard C library. Point it at a text file with one item per line, such as a list of names, and it learns the character patterns, then invents new entries that look like they belong. Forward pass, backpropagation, the Adam optimiser and sampling are all written by hand, with vector instructions for ARM and x86 chips.
Why it's ranked here
The case rests on measured numbers, not adjectives. The model has 4,192 parameters and posts nearly identical loss on 20,000 training names and 12,033 held-out ones, beating an interpolated trigram with about five times the parameters. Inference runs around ten million tokens per second single-threaded on an Apple M5 Pro and about seven million on a Ryzen 5600H. MIT licensed, and the performance notes explain exactly where each cycle goes.
What's good
The performance write-up is unusually honest. It reports the machine is limited by instruction issue width rather than stalls, backed by a measurement of 0.20 cycles per extra multiply-add against a 0.19 minimum. It lists abandoned experiments, like half-precision weights and refined reciprocals, with the measured result of each, so nobody repeats them. The fast inference path is checked against the training path, agreeing to within fp32 rounding noise. The build picks compiler flags for the host automatically.
Tradeoffs
It is a teaching and benchmarking object, not a usable language model: one layer, 16-wide embeddings and a 16-character context. It refuses to compile on anything other than 64-bit x86 with AVX2 or 64-bit ARM with NEON. The catalogue pitch leans on AVX2, but the docs say the x86 path is the less tuned one, with almost none of the ARM tuning repeated there. Input lines longer than 255 characters get split into separate entries, and there is no batching.
How to use it well
Read it if you want to see every step of GPT training and decoding in plain C with no framework hiding anything, or if you care about squeezing a small matrix-vector workload onto one CPU core. Feed it short line-per-item corpora like names or words. Benchmark on native hardware: the docs measure x86 emulation on Apple Silicon at a third of native speed. For real text generation or GPU training, look elsewhere.
Technical notes+
Everything lives in src/microgpt.c: N_EMBD 16, N_HEAD 4, N_LAYER 1, BLOCK_SIZE 16, a byte-level tokenizer built from the characters seen, xorshift RNG seeded at 42, and load_dataset reading up to 85000 lines through a 256-byte buffer. The build fails with #error on any architecture other than x86-64 or AArch64. docs/PERFORMANCE.md describes gpt_forward_infer: column-major packed weights, split accumulators, a precomputed (token, pos) table covering embedding, RMS norms and layer-0 Q/K/V, a deferred RMS scale that relies on ReLU squared being positively homogeneous, and fused sampling. The Makefile passes -O3 -march=native -ffast-math plus -mavx2 -mfma on non-ARM hosts. The file also defines a fast exp approximation, fexpf.
Observed
- License
- MIT
- Language
- C, single source file, links only libm
- Build
- Makefile with make and make run targets; flags chosen per host architecture
- Interface
- Command-line binary taking a one-item-per-line text corpus
- Platforms
- macOS, Linux, Windows via MSYS2; x86-64 with AVX2 or ARM64 with NEON only
- Model size
- 4,192 parameters, one layer, 16-dim embeddings, 16-token context
- Tests
- No test directory in the repository tree
Read from README.md, Makefile, LICENSE, src/microgpt.c, docs/PERFORMANCE.md.
What it can do
Train a transformer language model from scratch
Text training data → Trained GPT model weights
Generate text samples using trained model
Text prompt and trained model → Generated text continuation
Compile C code with AVX2 optimizations
Source code and compilation flags → Optimized executable binary
Run inference on transformer model
Model weights and input tokens → Predicted next tokens
Execute zero-dependency C implementation
Compiled binary → Standalone model training or inference
Tags
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.