- Category
- AI Tools
- Rank
- No. 326Tools index
- Listed in
- #5 Run models locally
- Type
- APP
- Builder
- jundot
- GitHub
- 21.8k stars
- Latest release
- v0.7.0.dev2
- Date
About
An LLM inference server for Apple Silicon with continuous batching and SSD caching.
What it does
oMLX turns an Apple Silicon Mac into a local model hub. It discovers models in configured directories, exposes them to existing clients through OpenAI-compatible and Anthropic-compatible endpoints, and provides browser and menu-bar controls. It can serve language, vision, OCR, embedding, and reranking workloads while automatically loading, pinning, or evicting models as memory changes.
Why it's ranked here
This is a strong choice for developers who want serious local model serving without surrendering operational control. The combination of broad model types, familiar client protocols, persistent prefix reuse, memory-aware model management, offline administration, and native Mac controls covers both daily convenience and demanding inference workflows. Its narrow hardware target and experimental features keep the verdict qualified.
What's good
The cache can retain reusable prompt prefixes in memory or safetensors on SSD, including across server restarts. One server handles text, vision, OCR, embeddings, and reranking. Operators get model pinning, idle timeouts, aliases, profiles, memory limits, live settings, benchmarking, downloads, chat, and monitoring. Existing clients can connect through established API shapes instead of custom adapters.
Tradeoffs
It requires Apple Silicon, macOS 15 or newer, and a supported Python release. Several model families need native kernels for acceptable performance, but source and Homebrew kernel builds require full Xcode; plain source installs silently use slower generic paths. Multi-Mac inference remains an experimental source-build preview. Experimental DFlash serving serializes requests, loads both target and draft models, and supports only selected model families.
How to use it well
Use it as the local inference backend for coding agents, private experiments, model comparison, or a small Mac-based service. Keep frequent models pinned, let heavier models unload automatically, and benchmark real prompts before enabling speculative paths. Prefer the packaged Mac application when native kernels matter. It does not replace a general cross-platform or hosted deployment layer.
Technical notes+
pyproject.toml defines a setuptools Python package, the omlx console script, FastAPI and Uvicorn dependencies, exact MLX pins, optional MCP, audio, grammar, ModelScope, and ParoQuant extras, plus pytest configuration targeting tests. setup.py conditionally builds four CMake-backed MLX kernel extensions when OMLX_WITH_CUSTOM_KERNEL or the matching build flag is present. omlx/cli.py initializes persisted settings, validates configuration, binds sockets before server initialization, and constructs cache limits. omlx/server.py exposes OpenAI completions, chat, Responses, embeddings and reranking models, Anthropic Messages compatibility, MCP routes, bearer or Anthropic-style API-key authentication, streaming, and model lifecycle state. omlx/oq.py implements sensitivity-driven mixed-precision quantization and validates model-specific constraints. docs/oQ_Quantization.md documents standard MLX safetensors output and streaming quantization. docs/distributed-cluster.md describes experimental unequal-memory pipeline partitioning with rank-local KV caches. docs/experimental/dflash_mlx_integration.md documents the separate speculative engine, fallback routing, and its concurrency and model-support limits. docs/CONTRIBUTING.md shows Python package, native SwiftUI app, packaging, tests, and documentation as distinct repository areas.
Observed
- License
- Apache 2.0
- Primary implementation
- Python core with a native Swift and SwiftUI macOS companion app
- Platform support
- Apple Silicon on macOS 15.0 or newer; Python 3.11 through 3.13
- Install surface
- macOS DMG application, Homebrew package and service, or editable Python source install
- Interfaces
- CLI, browser admin and chat UI, menu-bar app, OpenAI-compatible API, Anthropic Messages API, and optional MCP support
- Model workloads
- Text language models, vision-language models, OCR, embeddings, and rerankers
- Repository structure
- Separate main package, SwiftUI application, packaging pipeline, test suite, and documentation directories
Read from README.md, setup.py, pyproject.toml, docs/CONTRIBUTING.md, docs/oQ_Quantization.md, docs/distributed-cluster.md, docs/experimental/dflash_mlx_integration.md, omlx/oq.py, omlx/cli.py, omlx/server.py.
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.