Builder

LMCache

1 Tool

LMCache is an open-source KV cache management layer for large language model inference that grew out of systems research at the University of Chicago. It lets serving engines like vLLM and SGLang store, reuse, and share KV caches across GPU memory, CPU memory, and external storage backends such as Redis, S3, and local disk, reducing redundant computation on repeated or shared prompts. The project reports significant throughput and latency improvements and is used in production by companies including Google Cloud, AWS, NVIDIA, and IBM.