Inference Stack
www.inferencestack.io- Category
- Developer Tools
- Rank
- No. 4544Tools index
- Pricing
- Open Source
- Type
- TOOL
- Date
About
Inference Stack is an MIT-licensed bundle that deploys a self-hosted LLM serving setup with one install command, using Docker Compose on one machine or Helm on Kubernetes. It combines vLLM for running models on NVIDIA GPUs, LiteLLM for a single API with keys, teams, and budgets, llm-d for choosing a replica per request from cache and queue data, and Grafana dashboards for health and traffic. A simulation mode returns canned responses for testing without a GPU.
Intel on Inference Stack
Tags
llminferencevllmlitellmkubernetesself-hosted
Media

Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.