Vibeleaderboard

How do I run an LLM locally, and what hardware do I need?

Start with LM Studio for a desktop interface, Ollama for a local runner and API, or llama.cpp for runtime control. Budget memory for model weights, context cache, and overhead, then measure speed on your own hardware.

The runtime runs the model; it does not determine what that model can do. Pick a task and a candidate model before tuning your setup. If keeping prompts on your machine is the goal, also check the application’s provider settings and any connected tools: a local interface can still call a remote service.

Surveyed 10 September 2026

Estimate memory, then measure your workload

  1. Weight storage is roughly parameter count × bits per weight ÷ 8. For example, 8 billion parameters at 4 bits is about 4 GB in decimal units for weights alone. Quantization metadata, runtime buffers, and the context cache add to that estimate.
  2. Leave room for your operating system and other applications. CPU offload can make a model runnable when it exceeds GPU memory, but measure latency before committing to that setup.
  3. Test a short prompt and your longest realistic prompt. Record memory use, time to the first token, and whether the answer solves the task. Longer context can change both memory needs and speed.
  4. If an application expects an OpenAI-compatible API, test its actual requests. Similar endpoints do not guarantee identical support for tool calls, structured output, or every request option.

Run models locally

Open in Tools →
No.Tool
  1. 01
    Ollama

    Start here if you want a local model runner you can use from commands or connect to an application.

    Tradeoff: Context length affects memory use. Increase it for your workload deliberately and check whether the model still fits comfortably.

    Alternative: Choose LM Studio for a desktop-first setup or llama.cpp for lower-level control.

    Product documentation
    AI Tools
  2. 02
    llama.cpp

    Consider it when you need direct control over model files, quantization, hardware backends, and CPU/GPU inference.

    Tradeoff: That control means more configuration decisions. Matching the model format and build to your hardware is part of the setup.

    Alternative: Use Ollama or LM Studio when managing and running a model matters more than tuning the runtime.

    Product documentation
    AI Tools
  3. 03
    LM Studio

    Start here if you prefer a desktop interface and also want to serve a local model to other applications. Its developer tools include local server APIs.

    Tradeoff: Check the API features your client uses and the loaded model, rather than assuming every hosted-model integration will work unchanged.

    Alternative: Use Ollama for a command-oriented workflow or llama.cpp for direct runtime configuration.

    Product documentation
    AI Tools
  4. 04
    AirLLM

    Layer-by-layer memory management lets a single 4GB GPU run inference on 70B-parameter models.

    AI Tools
  5. 05
    omlx

    LLM inference server built for Apple Silicon, featuring continuous batching and SSD caching.

    AI Tools
  6. 06
    rllama

    Ruby FFI bindings for llama.cpp that run open models like GPT-OSS, Qwen, Gemma, and Llama 3 locally.

    AI Tools
  7. 07
    WhichLLM

    Benchmarks local LLMs on your specific hardware and ranks which models run best on it.

    Developer Tools
  8. 08
    DwarfStar (ds4)

    Local inference engine purpose-built for DeepSeek V4 and GLM 5.2 models with Metal, CUDA, and ROCm backends.

    Developer Tools
  9. 09
    RamaLama

    Serves AI models locally by pulling them as OCI containers and running hardware-accelerated inference, no Python setup.

    AI Tools
  10. 10
    vLLM

    An open-source LLM inference engine using PagedAttention and continuous batching to maximize GPU throughput.

    Developer Tools
  11. 11
    GPT4All

    A desktop app for running open-source LLMs locally on Windows, macOS, and Linux without a GPU or cloud API.

    AI Tools
  12. 12
    AnythingLLM

    An all-in-one, self-hostable AI app for chatting with documents and building custom agents without cloud APIs.

    AI Agents
  13. 13
    LocalAI

    A self-hostable AI runtime that runs LLM, vision, voice, image, and video models on any hardware behind one API.

    Developer Tools
  14. 14
    Jan

    An open-source desktop app for running LLMs locally or connecting to cloud providers like OpenAI and Anthropic.

    AI Tools
  15. 15
    llamafile

    Packages an LLM's weights with llama.cpp into a single cross-platform executable that runs with no installation.

    Developer Tools
  16. 16
    KoboldCpp

    A single-file inference engine built on llama.cpp that runs GGUF models locally with bundled multimodal features.

    AI Tools

A curated selection in editorial order. Use the fit and evidence to judge it for your task. Something missing?

What to look for

  • 01Does the model fit with room for context, runtime overhead, and other applications? Check memory during a realistic prompt, not only while loading the weights.
  • 02Does the runtime support the API features your application needs? Test tool calls and structured outputs explicitly if you rely on them.
  • 03Does the chosen quantization preserve quality on your tasks? Compare outputs from the same model at different precisions when memory allows.

Common questions

Can a local model replace a frontier model for coding?
It depends on the model, task, hardware, and agent around it. Test a representative coding change and review the result. A model that handles autocomplete well may still struggle with your multi-file bug fix.
What is quantization and what does it cost me?
Quantization stores model weights at lower precision to reduce their memory footprint. The quality and speed tradeoffs depend on the model, quantization method, hardware, and task. Smaller weight files also leave more memory for context.

More in Understand the AI landscape