llama.cpp
github.com/ggml-org/llama.cpp- Category
- AI Tools
- Rank
- No. 11Tools index
- Listed in
- #2 Run models locally
- Pricing
- Open Source
- Type
- TOOL
- Builder
- ggml-org
- GitHub
- 127.5k stars
- Latest release
- b10867
- Date
About
A C/C++ library for running large language models locally with minimal setup and optimized performance across different hardware architectures. Enables LLM inference on CPUs and GPUs with various quantization options to reduce memory usage.
What it does
llama.cpp takes models stored in GGUF format and exposes them through an interactive command line, a built-in web interface, an OpenAI-compatible server, or a C-style library. It can download compatible models directly, run local model files, and divide computation between processors and graphics hardware.
Why it's ranked here
The project combines an unusually broad hardware matrix with practical entry points for both individuals and applications. Its strongest case is deployment flexibility: one model runner spans desktop processors, several GPU families, mobile builds, containers, and mixed processor plus GPU execution. That breadth comes with uneven backend capabilities and configuration work.
What's good
Hardware support goes beyond nominal compatibility. The project documents architecture-specific processor instructions, dedicated graphics kernels, hybrid execution, and multiple-GPU strategies. Quantization ranges from 1.5-bit through 8-bit integers. Users can start with packaged binaries or containers, while integrators get a C-style library and an OpenAI-compatible HTTP server.
Tradeoffs
Models must use GGUF, so other formats require conversion. Backend support is not uniform: the operation matrix contains partial and missing implementations. Tensor-parallel multi-GPU execution remains experimental, excludes several model architectures, and disables automatic memory fitting. GPU-enabled container images are built but not otherwise tested by continuous integration.
How to use it well
Choose it when you need one inference core across personal machines, servers, containers, or embedded applications. Start with a compatible GGUF model and the command line, then adopt the HTTP server or library for integration. Treat backend selection and memory tuning as deployment engineering. It does not supply model training or model creation.
Technical notes+
README.md defines the CLI, built-in web UI, OpenAI-compatible server, quantization range, and backend matrix. docs/build.md makes CMake the supported build path and identifies the main product as a library with a C-style interface; Makefile deliberately aborts and redirects builders to CMake. docs/models.md requires GGUF and describes repository Python converters for other formats. pyproject.toml packages those conversion scripts for Python 3.10 through 3.14 with NumPy, SentencePiece, Transformers, Protobuf, Torch, and the local GGUF package. docs/ops.md records operation support per backend, including partial and absent implementations. docs/docker.md separates full, light, and server images and states that GPU images receive no CI testing beyond successful builds. docs/multi-gpu.md documents layer, deprecated row, and experimental tensor split modes, including tensor-mode architecture and memory-fitting constraints. docs/llguidance.md makes LLGuidance optional behind LLAMA_LLGUIDANCE, requires Rust and Cargo, and notes that its errors go to stderr while generation continues.
Observed
- License
- MIT
- Primary language
- C and C++
- Interfaces
- C-style library, command-line tools, built-in web UI, and OpenAI-compatible HTTP server
- Installation
- Pre-built binaries, Docker images, source builds with CMake, conda-forge, Winget, Homebrew, MacPorts, and Nix
- Model format
- Inference requires GGUF; repository Python scripts convert other model formats
- Platform support
- Documented targets include Windows, macOS, Linux, Android, Apple silicon, x86, RISC-V, several GPU families, and multiple accelerator backends
- Packaging
- Python conversion scripts require Python 3.10 through 3.14 and are packaged with Poetry
- Build system
- CMake is supported; the root Makefile intentionally stops with migration guidance
Read from README.md, Makefile, pyproject.toml, requirements.txt, docs/ops.md, docs/build.md, docs/docker.md, docs/models.md, docs/preset.md, docs/android.md, docs/install.md, docs/release.md, docs/multi-gpu.md, docs/autoparser.md, docs/llguidance.md.
What it can do
Run large language models locally
LLM model file and text prompt → Generated text response
Perform LLM inference on CPU
Quantized model and user query → Model predictions and responses
Perform LLM inference on GPU
Model file and input text → Accelerated text generation
Quantize language models
Original LLM model → Compressed model with reduced memory usage
Optimize model performance across hardware architectures
Model file and hardware specifications → Hardware-optimized model execution
Execute LLM inference with minimal setup
Model file and configuration parameters → Ready-to-use local LLM service
Intel on llama.cpp
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.