LMDeploy

Why it's ranked here

OpenMMLab built this to cover ground that CUDA-only servers skip: the same install targets Ascend, Cambricon and MetaX accelerators alongside Nvidia GPUs, docs list Llama, Qwen, InternLM, DeepSeek, GLM, Mixtral and a matching column of vision-language models, and every model routes through one of two engines, a compiled one built for throughput and a pure Python one built so new architectures land without a rebuild. Few open serving projects carry that hardware breadth and model breadth at once, which is why this belongs on a shortlist even before its speed claims are checked independently.

What's good

The two-engine split is well thought out rather than bolted on: the compiled backend reports itself unavailable rather than crashing when its extension isn't built, so the pure Python engine keeps working without it. The command line tool exposes the details that matter for production instead of hiding them behind defaults, quantization policy per request, tensor and expert parallelism for mixture-of-experts checkpoints, and prefix caching. Model coverage is unusually wide for a serving project, spanning ordinary chat models up to trillion-parameter mixture-of-experts systems and their vision-capable counterparts.

Tradeoffs

Only the compiled engine gets full performance, and building it needs a working cmake and pybind toolchain, a compatible CUDA release, and an actual Nvidia GPU; the pure Python engine is the fallback everywhere else, including plain CPU boxes. Even a single-file offline script pulls in the full serving stack, since the top-level import wires the client, pipeline and server together rather than keeping them separate. The extensive model list also only says a model is wired in, not that it is fast: the newest, largest entries assume multi-GPU setups this review cannot verify.

How to use it well

Reach for this when serving a model in production on real GPU hardware, especially Ascend, Cambricon or MetaX chips where fewer serving frameworks even try, or when quantizing a model's weights and cache to fit tighter memory. The command line covers both engines directly, so start there to pick tensor parallelism, expert parallelism for mixture-of-experts checkpoints, and a quantization policy before writing any Python. For quick offline scoring or batch generation without standing up a server, the library's pipeline interface is the shorter path; skip the compiled engine entirely if there's no CUDA toolchain to build it.

Observed

Packaging

Installable via pip from PyPI; the compiled engine additionally requires a cmake and pybind11 build toolchain and a CUDA-capable environment.

Engine options

Ships two distinct inference engines in one package: a compiled engine and a pure-Python engine, selected per run.

Interfaces

Exposes three interfaces: a command-line tool with subcommands, a Python library (pipeline, serve, client), and a documented API server.

Platform support

Includes get-started documentation for non-Nvidia accelerator platforms (Ascend, MetaX, Cambricon) alongside the default CUDA path.

Python support

Declared Python compatibility spans 3.10 through 3.13.

CLI options

The command-line chat subcommand exposes tensor-parallel, expert-parallel, prefix-caching, and speculative-decoding arguments directly.

Runtime checks

The compiled engine's absence is handled with a runtime availability check rather than an import failure.