llama2.c

Why it's ranked here

This is a strong teaching and experimentation tool because training, checkpoint conversion, tokenization, sampling, and inference sit in one compact repository. Small models can run interactively, while the readable implementation exposes the full computation. Its value drops for large models, where float32 storage and CPU execution become slow and memory-heavy.

What's good

The project connects model training directly to a small native inference engine. It supports prompts, chat mode, temperature, top-p sampling, grouped multiquery attention, and memory-mapped checkpoints. The quantized path keeps sensitive normalization parameters in float32 while moving matrix-multiplication weights and activations to int8. Build targets cover basic, debug, optimized, multithreaded, and Windows configurations.

Tradeoffs

The default inference path stores every weight in float32, producing large checkpoints and poor practicality beyond seven billion parameters. Quantization reduces size and can improve speed, but adds runtime conversion overhead and gives up some numerical correctness. Code Llama support is explicitly incomplete and buggy because some architecture constants differ. Meta checkpoints also require separate access and conversion.

How to use it well

Use it to learn transformer inference, validate exported checkpoints, or train narrow small models before running them locally. Start with the supplied tiny-model workflow, compare deterministic and top-p sampling, then inspect the quantized path to understand performance tradeoffs. Do not choose it for large-model throughput, broad architecture compatibility, or a ready-made network serving API.

Observed

Primary languages

C for native inference and Python for training, export, tokenization, and sampling workflows.

Install surface

Native binaries compile through Make; Python dependencies are pinned in requirements.txt.

Interface

Command-line programs accept checkpoints, prompts, sampling controls, tokenizer paths, and chat mode.

Checkpoint formats

Supports legacy float32, header-based float32, and grouped Q8_0 int8 exports.

Platform support

Build targets cover GCC or Clang environments, OpenMP builds, GNU-compatible systems, and Windows cross-compilation.

Testing

Make targets run pytest and a standalone C tokenizer test.