Ollama
github.com/ollama/ollama- Category
- AI Tools
- Rank
- No. 15Tools index
- Listed in
- #1 Run models locally
- Pricing
- Open Source
- Type
- TOOL
- Builder
- ollama
- GitHub
- 180.5k stars
- Latest release
- v0.34.0-rc3
- Date
About
An open-source tool that lets you run and manage large language models locally on your machine. It supports popular models like Llama, Gemma, DeepSeek, and Qwen, providing both command-line interface and REST API for easy integration.
What it does
Ollama is a single Go program that runs large language models on your own machine. A command-line client talks to a long-running local server over HTTP, and that server hands the actual inference work to a separate open-source engine running as a subprocess. A customised model variant is just a short text file naming the base model, its parameters, and its prompt template, so tailoring a model takes one file and one command rather than a fork. A newer built-in agent runtime adds tool calling, a skills catalog, approval prompts, and automatic context compaction.
Why it's ranked here
Ollama owns the unglamorous part of local models: fetching weights, fitting them to your hardware, keeping a process warm, and exposing one stable local HTTP interface that every other tool can target. It does that across macOS, Windows, Linux, and Docker with seven named GPU build targets, and its dependencies track current versions rather than drifting. The public client library is the same one the product itself consumes, which keeps it honest, and the agent runtime shows the project extending upward into tool orchestration instead of standing still.
What's good
The API is shaped by people who actually consume it. Thinking can be unset, off, or set to a level, and those states are distinguishable. When a conversation outgrows the context window you choose between two named behaviours, truncating or shifting, instead of hitting a single error. Token probabilities are first class, and tool-call arguments keep the exact field order the model produced when written back out. Before logging, the server blanks any environment value whose name looks like a secret, and the agent loop caps tool output and retries a failing model twice by feeding the error back as a turn.
Tradeoffs
Generation quality and kernel speed belong to the underlying inference engine, so a whole class of bugs is inherited rather than fixable here. Building from source is not a plain Go build: it needs CMake, a C and C++ compiler, and explicit flags per GPU family, and the server only runs against an already-built native payload. The default client applies no request timeout, so cancelling a hung call is entirely your responsibility, and it reads whole response bodies into memory. The in-repo API reference is a partial mirror; the canonical version now lives on the hosted documentation site.
How to use it well
Program against the official Go, Python, or JavaScript client libraries rather than raw HTTP, and always pass a deadline, because the client will never time out on its own. Set the context window deliberately: the default is a small 2048 tokens, and a request above the model's trained length is silently lowered. Prefer the named shift or truncate overflow behaviours before letting long conversations fail. For tool-using apps, use the chat endpoint with tools declared, and tune the keep-alive setting so the model is not reloaded on every request. When building from source, list your GPU backends explicitly instead of relying on autodetection.
Technical notes+
main.go is a four-line handoff to a cobra CLI that talks to a daemon on port 11434. api/client.go is the same client the CLI consumes: unary calls go through do(), streaming calls through stream(), which scans NDJSON with an 8 MB per-line ceiling and surfaces mid-stream errors as StatusError or AuthorizationError. ClientFromEnvironment returns a client backed by http.DefaultClient, which sets no request timeout, and do() reads whole bodies with io.ReadAll. When the target host is ollama.com or auth is enabled, the client builds a challenge of the form METHOD,path?ts=unix, signs it via auth.Sign, and sends it as an Authorization header with a ts query parameter. llm/server.go declares the LlamaServer interface; NewLlamaServer serves all GGML models through an upstream llama-server subprocess, clamps a requested num_ctx above the model's trained context length with a warning, and redacts environment values whose key contains KEY, TOKEN, SECRET, PASSWORD or CREDENTIAL before logging. agent/session.go implements the agent runtime: tool registry, skills catalog, approval prompting, context compaction, a default of 100 consecutive model/tool cycles, a 60,000-rune tool-output cap reduced to 6,000 and 3,200 for small and tiny context windows, and up to two retries on consecutive 5xx model errors fed back as user turns. Modelfile directives are FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER, MESSAGE and REQUIRES, with a default num_ctx of 2048 per docs/modelfile.mdx; docs/development.md requires CMake 3.24 or newer and tests run with go test.
Observed
- Language and packaging
- Go; one binary providing the CLI, the local HTTP server, and the client library
- Interfaces
- CLI, streaming HTTP API on localhost port 11434, Go client, published Python and JavaScript libraries
- Inference backend
- llama.cpp; GGML models run in an external llama-server subprocess
- Platform support
- macOS, Windows, Linux, and Docker, with seven named GPU build targets
- Build surface
- Building from source requires CMake and a C/C++ compiler in addition to Go
Read from README.md, go.mod, main.go, api/client.go, api/types.go, llm/server.go, docs/api.md, docs/quickstart.mdx, docs/modelfile.mdx, docs/development.md, agent/session.go.
What it can do
Run large language models locally
Model name and user prompts → AI-generated text responses
Download and install language models
Model name (e.g., Llama, Gemma, DeepSeek, Qwen) → Locally installed model files
Manage multiple language models
Model management commands → Updated model registry and storage
Process API requests for text generation
HTTP requests with prompts via REST API → JSON responses with generated text
Execute models via command line
Command line instructions and text prompts → Text responses in terminal
Deploy models in Docker containers
Docker configuration and model specifications → Containerized model instances
Intel on Ollama
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.