Vibeleaderboard

How do I run an LLM locally, and what hardware do I need?

Memory is the constraint. A 7B model quantized to 4-bit needs roughly 5GB, a 70B closer to 40GB, and it must fit in unified or GPU memory to run at usable speed. Runtime choice matters far less.

Surveyed 10 August 2026

Run models locally

Open in Tools →
No.Tool
  1. 01
    WhichLLM

    Benchmarks and ranks which local LLM runs best on your specific hardware.

    Developer Tools
  2. 02
    llama.cpp

    A C/C++ library for running large language models locally with minimal setup and optimized performance across different hardware architectures. Enables LLM inference on CPUs and GPUs with various quantization options to reduce memory usage.

    AI Tools
  3. 03
    AirLLM

    A library that runs 70B-parameter LLM inference on a single 4GB GPU through aggressive layer-by-layer memory management.

    AI Tools
  4. 04
    rllama

    Ruby FFI bindings for llama.cpp to run open-source LLMs like GPT-OSS, Qwen, Gemma, and Llama 3 locally with Ruby.

    AI Tools
  5. 05
    omlx

    An LLM inference server for Apple Silicon with continuous batching and SSD caching.

    AI Tools
  6. 06
    p1

    LLM-based code completion engine from the ggml team — local-first autocomplete for editors using small models.

    AI Tools

Ordered by the VibeLeaderboard index, re-surveyed each edition. Something missing?

What to look for

  • 01Does the model fit in your memory at your chosen quantization? Everything else is secondary.
  • 02Does the runtime expose an OpenAI-compatible endpoint? That decides whether existing code works unchanged.
  • 03How much quality does the quantization cost? Below 4-bit, degradation becomes noticeable on reasoning tasks.

Common questions

Can a local model replace a frontier model for coding?
Not yet for hard multi-file work. Local models are genuinely useful for autocomplete, classification, summarization, and anything where privacy or per-call cost outweighs peak capability.
What is quantization and what does it cost me?
Storing the model's weights at lower precision to shrink memory use. 8-bit is near-lossless, 4-bit is the usual quality-to-size sweet spot, and below that reasoning degrades noticeably.

More in Understand the AI landscape