gemma.cpp
https://github.com/google/gemma.cpp- Category
- AI Tools
- Rank
- No. 626Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- GitHub
- 7.0k stars
- Latest release
- v0.1.4
- Date
About
Google's lightweight, standalone C++ inference engine for Gemma models — runs Gemma on CPU with no dependencies, no GPU required.
What it does
A model-specific runtime that exposes Gemma 2, Gemma 3, and PaliGemma 2 through an interactive terminal, streaming C++ interfaces, and basic Python bindings. Its compact core keeps model execution visible and modifiable, while portable SIMD selects processor instructions at runtime.
Why it's ranked here
The project makes a strong research tool because it combines a small, direct implementation with serious CPU machinery: compressed matrix multiplication, runtime autotuning, tensor parallelism, and streaming inference. Its deliberately narrow model scope improves inspectability, but limits general-purpose deployment value.
What's good
It supports text and vision-language models, single-query and batched streaming, sampling, a backward pass, and an Adam optimizer. Weight compression is integrated into matrix multiplication across several numeric formats. Portable SIMD, memory-mapped or parallel reads, and processor-aware threading address real CPU bottlenecks.
Tradeoffs
The project prioritizes experimentation over production edge deployment, and its maintainers recommend mainstream Python frameworks for that job. Model coverage is intentionally narrow. Safetensors conversion is not fully open sourced. Vision input accepts only binary PPM images, and long contexts remain slow because attention cost grows quadratically.
How to use it well
Choose it for studying, modifying, or embedding Gemma inference when low-level CPU behavior matters. Start with the recommended small instruction-tuned compressed checkpoint, then use the terminal for exploration or the C++ interfaces for integration. Do not treat it as a general model runtime or a production edge deployment stack.
Technical notes+
README.md describes a roughly 2K-line core plus roughly 4K lines of utilities, CMake and Bazel builds, C++17 requirements, pybind11 bindings, and Linux, Windows, and OS X support. gemma/run.cc implements interactive and single-prompt execution, token streaming, conversation reset, KV-cache reuse, and optional image-token generation. gemma/gemma.cc dispatches model work through Highway targets. gemma/vit.cc implements vision-transformer attention and image patch processing. ops/matmul.cc generates cache-aware runtime autotuning candidates. io/io.cc uses POSIX pread, preadv, mmap, and advisory sequential access where available, while io/io_win.cc supplies Windows file handles and mapped views. io/fields.cc implements bounded, forward-compatible metadata serialization. ops/dot_test.cc contains numerical dot-product tests.
Observed
- License
- Apache License 2.0
- Primary language
- C++ with a C++17 minimum
- Build surface
- CMake and Bazel source builds
- Interfaces
- Interactive CLI, streaming C++ APIs, and basic pybind11 Python bindings
- Platform support
- Linux, Windows, and OS X
- Model support
- Gemma 2, Gemma 3, and PaliGemma 2
- Acceleration
- CPU SIMD through Google Highway with runtime instruction-set selection
- Tests
- Repository text includes a C++ numerical dot-product test source
Read from README.md, docs/CONTRIBUTING.md, io/io.cc, gemma/run.cc, gemma/vit.cc, io/fields.cc, io/io_win.cc, ops/matmul.cc, gemma/gemma.cc, ops/dot_test.cc.
What it can do
Run Gemma model inference on CPU
Gemma model files and text prompts → Generated text responses
Load Gemma models for local execution
Gemma model files → Loaded model ready for inference
Process text prompts without GPU
Natural language text prompts → AI-generated text responses
Execute standalone inference without dependencies
Text queries and Gemma model → Language model predictions
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.