Vibeleaderboard
Index / tool
Visit github.com
Category
AI Tools
Rank
Pricing
Open Source
Type
TOOL
Use case
Models: Train & Run
Interfaces
API
Latest release
v0.2.15
Date

About

Strata is an inference engine that runs the Qwen3.8-Flash-Next model on consumer hardware, with one-click installers for Windows and Linux. It serves an OpenAI and Anthropic compatible API on localhost and supports optional image input.

What it does

Strata puts a 125-billion-parameter mixture-of-experts model on a desktop with a 12 GB graphics card. The trick is splitting the work: the card keeps the few thousand experts used most often and learns which ones those are as you chat, system memory holds all 24,576 of them while the processor computes the misses in parallel, and a 29 GB lookup table stays on the SSD. A small draft layer guesses up to three tokens that the full model then checks in one pass, so output is unchanged but arrives 1.6 to 1.8 times sooner. Your apps and coding agents talk to it over a local HTTP endpoint.

Why it's ranked here

The numbers are the case. On an RTX 5070 with 64 GB of RAM the fastest compression writes 94 tokens a second and reads a 32K-token prompt at 2,650 tokens a second; an RX 9070 XT manages 60. Those are figures for a model the project says normally needs a server with hundreds of gigabytes of graphics memory. It ships under MIT, covers both NVIDIA and AMD cards, pins every model download to a fixed commit, and documents its security posture plainly, including that no outside audit has happened yet.

What's good

Setup recommends rather than forces. It reads your card, RAM and disk, proposes a model size, and keeps a bigger choice if you insist. Downloads resume, and nothing finished is redone on the next run. The local server rejects a prompt that overflows the context with an error instead of silently truncating it. Without an API key it refuses requests whose Host or Origin it does not recognise, which blocks DNS rebinding and cross-site pages, and those checks have their own test suite. An MCP (Model Context Protocol) server lets an assistant install, start, stop and benchmark it, but only on localhost and only after an explicit confirm.

Tradeoffs

The hardware bar is real: 32 GB of RAM minimum, 64 GB to run every size, about 80 GB of disk, and a 66 to 111 GB download. The first start loads 35 to 55 GB into memory and can freeze the PC for one to three minutes. By default it serves one request at a time; parallel serving slows each answer on a 12 GB card. The coding variant that fits 32 GB drops half the experts and gave wrong or looping answers in Chinese. AMD cards on Windows cannot read images yet, and calibration is NVIDIA-only. It runs one model family, not arbitrary weights.

How to use it well

Pick it if you want a large open model behind your existing coding agent or chat client with nothing leaving the machine, and you have a gaming PC with 64 GB of RAM. Point an OpenAI-compatible or Anthropic-compatible client at the local address; any key and model name work. Take the recommended two-bit size first and move up only if RAM allows. Set an API key before exposing it to your network. It does not cover serving many users at once or swapping in models from other families.

Technical notes+

setup.py drives install: it pins Hugging Face repos to fixed commits in HF_REVISIONS, builds against a pinned llama.cpp commit, fetches a prebuilt CUDA 13 engine (driver 580+) or compiles one, with a CUDA 12.9 path for Pascal and Volta. MODELS lists sizes Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, IQ1_M (Coder), UD-IQ4_XS and UD-Q4_K_XL with RAM and download figures. serve/server.py is a ThreadingHTTPServer exposing /v1/chat/completions, /v1/messages and more behind a FIFO, with an Engine protocol, a MockEngine for GPU-free tests, a 256-token repeat stop (REPEAT_STOP_TOKENS) and a 300 s silence watchdog (ENGINE_SILENCE_S). serve/test_security.py covers host_allowed and origin_allowed. requirements.txt pins Python dependencies; docs/MCP_SERVER.md describes a standard-library stdio MCP server that records process id and start time before ending any process.

Observed

License
MIT (LICENSE); model files carry their own licenses (docs/HOW_IT_WORKS.md)
Language
Python setup and HTTP server (setup.py, serve/server.py), engine built from parts of llama.cpp / ggml
Platforms
Windows 10/11 and Linux; NVIDIA RTX 20 to 50 series or listed AMD Radeon cards, 12 GB VRAM or more (README.md)
Install surface
One-click batch installer on Windows, shell installer on Linux, Docker image for NVIDIA on Linux (docs/INSTALL.md)
API
OpenAI chat completions and Responses, Anthropic messages, on 127.0.0.1:8080 by default (README.md, serve/server.py)
MCP
Stdio MCP server to install, start, stop and benchmark Strata (docs/MCP_SERVER.md)
Tests
Host and Origin check tests against a mock engine (serve/test_security.py)
Security audit
No outside security audit yet, per SECURITY.md

Read from README.md, requirements.txt, setup.py, LICENSE, docs/HOW_IT_WORKS.md, docs/MODELS.md, docs/INSTALL.md, docs/MCP_SERVER.md, serve/server.py, SECURITY.md, docs/TROUBLESHOOTING.md, serve/test_security.py.

Tags

llminferencelocal-llmqwenopenai-apiself-hosted

Tech Stack

PythonDocker

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.