LMCache
github.com/lmcache/lmcache- Category
- AI Tools
- Rank
- No. 76Tools index
- Pricing
- Open Source
- Platform
- cli
- Type
- TOOL
- Builder
- lmcache
- GitHub
- 11.8k stars
- Latest release
- nightly-rocm
- Date
About
A KV cache management layer that accelerates LLM inference by turning temporary cache into reusable knowledge that persists across sessions. Reduces time-to-first-token and improves throughput for long-context and multi-turn conversations.
What it does
LMCache runs beside LLM serving engines and manages their computed attention state. It can move that state from GPU memory into CPU RAM, local disks, or remote storage, then share it among requests and engine instances. It also supports cache transfer between prefill and decode workers, non-prefix reuse, custom transformations, and operational metrics.
Why it's ranked here
LMCache tackles a costly serving problem with unusually broad infrastructure support. Its standalone process survives inference-engine failures, while tiered storage and pluggable transports cover both single-node offload and distributed sharing. The strongest case is production LLM serving with repeated or long contexts. The alpha classification and substantial GPU infrastructure requirements make it a specialist choice, not a default dependency.
What's good
The storage abstraction spans CPU RAM, SSDs, Redis or Valkey, S3-compatible stores, and several specialized backends. CacheBlend can reuse blocks beyond a prompt prefix while selectively recomputing tokens to recover quality. Operators also get health, performance, cache-hit, lifecycle, and per-user usage metrics. Prefill and decode workers can exchange cache data over NVLink, RDMA, or TCP.
Tradeoffs
The package targets POSIX Linux, GPU environments, and a narrow supported Python range. Its build system includes native extensions and pins PyTorch during isolated builds, so source installation is more involved than a typical Python library. Hardware paths differ: automatic zero-copy transfer is CUDA-oriented, while other devices default to copied transfers unless they explicitly support handle exchange. The project metadata still labels development status as alpha.
How to use it well
Use LMCache when you operate LLM serving engines with repeated context, multi-turn agents, retrieval-augmented prompts, or separated prefill and decode workers. Start with measurable cache-hit and latency targets, then choose storage tiers and transports that match reuse distance and network topology. Treat it as serving infrastructure beside an inference engine. It does not replace the engine that loads models and generates tokens.
Technical notes+
pyproject.toml defines a setuptools package for Python 3.10 through 3.13, Linux classifiers, three console scripts, dynamic dependencies, and manylinux wheels for x86-64 and AArch64. setup.py selects platform-specific extension profiles and reads common plus profile-specific requirements. lmcache/__init__.py detects the active device backend and installs a compatibility shim that forwards operations to a resolved singleton. lmcache/sdk/__init__.py exposes KV and query-cache connections using ZMQ and HTTP endpoints. operator/cmd/main.go implements a Kubernetes controller manager with reconcilers, admission webhooks, health probes, leader election, and protected metrics.
Observed
- License
- Apache License 2.0
- Languages
- Core package and CLI are Python; the Kubernetes operator is Go.
- Installation
- Published as the lmcache Python package and installable with pip.
- Interfaces
- Python SDK plus lmcache, lmcache_server, and lmcache_controller console commands.
- Platform
- POSIX Linux, CPython 3.10 through 3.13, with GPU-oriented packaging.
- Build system
- Setuptools with platform build profiles, native extensions, and manylinux wheels for x86-64 and AArch64.
Read from README.md, setup.py, pyproject.toml, docs/README.md, docs/coding_standards.md, docs/design/README.md, docs/design/ARCHITECTURE_MULTI_HARDWARE.md, lmcache/__init__.py, setup_extensions/__init__.py, lmcache/cli/main.py, docs/source/conf.py, operator/cmd/main.go, lmcache/v1/__init__.py, lmcache/cli/__init__.py, lmcache/sdk/__init__.py.
What it can do
Manage KV cache for LLM inference
LLM key-value cache data → Optimized cache management layer
Enable persistent cache reuse across sessions
Temporary KV cache from previous sessions → Reusable persistent cache storage
Reduce time-to-first-token for LLM responses
LLM inference requests with cached context → Faster initial token generation
Improve throughput for long-context conversations
Long-context LLM inference workloads → Enhanced processing throughput
Optimize multi-turn conversation performance
Multi-turn dialogue requests → Improved conversation response times
Accelerate agentic workload processing
Agent-based LLM inference tasks → Faster agentic workflow execution
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.