Modelship

About

Modelship is an open-source system that starts a Ray cluster, deploys models through loaders such as vLLM, llama.cpp, diffusers and sherpa_onnx, and serves them behind an OpenAI Responses-compatible API. The author reports 17/17 on the Open Responses conformance suite, a Redis-backed response store, CPU, Metal and CUDA support, Prometheus metrics with a Grafana dashboard, automatic context sizing and MCP support. It runs under Docker, Kubernetes or a native install.