rvLLM Serverless
github.com/instructkr/rvllm-serverless- Category
- AI Tools
- Rank
- No. 1676Tools index
Previous survey · No. 1570 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- instructkr
- GitHub
- 40 stars
- Date
About
Lightweight, instant-startup vLLM replacement for RunPod serverless environments — cold-start in seconds instead of minutes.
What it does
rvLLM Serverless wraps a Rust inference server as a queue-based RunPod worker. It starts the local backend, waits for a healthy response, translates simple chat or completion jobs into OpenAI-compatible requests, and returns regular or streamed results through RunPod’s endpoint APIs.
Why it's ranked here
The design is focused and credible: the wrapper delegates inference, supports runtime-loaded and image-baked models, and has reached real GPU workers. The repository still labels itself work in progress, while automated coverage concentrates on configuration and request translation rather than proxy streaming, process supervision, or full deployment behavior.
What's good
The thin boundary is the main strength. Configuration stays environment-driven, model names can remain stable when a baked model uses a local target, and callers may use concise chat payloads or explicit proxy requests. Health-gated startup, streamed responses, private-model tokens, deterministic image pinning, and packaged GPU kernels address practical deployment concerns.
Tradeoffs
This is tightly scoped to queue-based RunPod GPU deployments and does not expose the container port directly. Building requires a sibling checkout of the underlying inference project, plus Docker build tooling. Error handling exits the worker on CUDA-related failures, and the documented tests do not cover streaming, backend failures, health-check timeouts, or end-to-end RunPod execution.
How to use it well
Use it when you already want rvLLM inference behind RunPod’s job API and prefer a small adapter over another inference layer. Start with the published generic image for runtime model selection, then bake frequently used models when image-contained snapshots suit deployment. It does not replace model evaluation, capacity planning, inference tuning, or a general-purpose HTTP hosting platform.
Technical notes+
src/handler.py creates ServerlessConfig, starts RvllmServerLauncher, registers the asynchronous RunPod handler, and aggregates streamed output. src/request_mapping.py infers chat, completion, model-list, or explicit proxy routes, injects the served model name, and rewrites that name to and from the local model target. src/proxy.py uses httpx.AsyncClient, parses server-sent data: lines, and stops on [DONE]. src/server_launcher.py launches the rvllm serve subprocess and polls /health. Dockerfile builds rvLLM with CUDA support, compiles and copies PTX kernels, installs the Python wrapper dependencies, and optionally runs builder/download_model.py to bake a Hugging Face snapshot. scripts/build.sh constructs a linux/amd64 Buildx context from this repository and a sibling rvllm checkout. Tests in tests/test_config.py and tests/test_request_mapping.py cover configuration defaults, launch arguments, route inference, validation, and model-name rewriting.
Observed
- Primary language
- Python implements the serverless wrapper, configuration, request mapping, proxy, launcher, and tests.
- Packaging
- Distributed as a multi-stage Docker image with a Rust rvLLM binary, compiled PTX kernels, and a Python virtual environment.
- Build surface
- A shell build script uses Docker Buildx and requires a sibling rvllm source checkout.
- Interface
- Queue-based RunPod endpoint API accepting direct OpenAI-style chat or completion inputs and explicit proxy requests.
- Model sources
- Supports Hugging Face model identifiers at runtime and model snapshots baked into the container image.
- Platform
- RunPod GPU workers using a Linux AMD64 container and CUDA 13.0-compatible configuration.
- Tests
- The tests directory contains unittest coverage for configuration and request mapping.
Read from README.md, src/__init__.py, src/proxy.py, src/config.py, src/handler.py, src/request_mapping.py, src/server_launcher.py, builder/download_model.py, tests/test_config.py, tests/test_request_mapping.py, Dockerfile, .runpod/hub.json, scripts/build.sh, scripts/smoke_test.sh, builder/requirements.txt.
What it can do
Deploy language models with instant startup
Language model configuration → Running model instance
Replace vLLM deployment in serverless environments
vLLM configuration and model → Lightweight serverless deployment
Reduce cold-start time for model inference
Model deployment request → Active model instance in seconds
Run language model inference on RunPod serverless
Text prompts and model parameters → Generated text responses
Scale language model deployments automatically
Traffic load and scaling parameters → Dynamically scaled model instances
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.