LiteLLM
github.com/berriai/litellm- Category
- AI Tools
- Rank
- No. 30Tools index
- Pricing
- Free
- Type
- TOOL
- Builder
- berriai
- GitHub
- 56.2k stars
- Latest release
- v1.96.2
- Added
- Jun 28, 2026
About
Open-source AI Gateway providing a unified, OpenAI-compatible interface to call 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, and more). Ships as a Python SDK and a deployable proxy server with virtual keys, spend tracking, load balancing, and guardrails.
What it does
LiteLLM sits between an application and model vendors, translating one request style across many provider implementations. Teams can embed it in Python or run it as a shared service. Its surface extends beyond chat into embeddings, images, audio, batches, reranking, agents, MCP tools, OCR, retrieval pipelines, and evaluations.
Why it's ranked here
The strongest case is operational breadth backed by real implementation depth. Provider-specific handlers cover many model and media types, while the gateway adds centralized routing and administration. The repository also shows serious engineering controls: extensive test targets, strict type checking, formatting gates, circular-import checks, and import-safety checks. The cost is a large system with substantial configuration and dependency weight.
What's good
It supports direct library calls and network clients that already speak the OpenAI protocol. The gateway exposes broad endpoint coverage, including responses, embeddings, images, audio, batches, reranking, agents, and MCP. Separate gateway and dashboard backends can trim one FastAPI application into data-plane and management surfaces. Optional extras keep several specialized features outside the base installation.
Tradeoffs
The base package already depends on HTTP clients, tokenizers, schema tooling, AWS libraries, and the OpenAI client. Running the proxy adds a much larger server, authentication, storage, scheduling, cryptography, and observability stack. Provider capability varies by endpoint, so a common request shape does not mean every vendor supports every operation. Some proxy features also depend on separately packaged LiteLLM components.
How to use it well
Use it when one application or team must switch providers, centralize model access, or expose a familiar API to existing clients. Start with the Python package for local integration, then adopt the proxy when shared keys, routing, budgets, or administration justify another service. Treat endpoint support as provider-specific and test your chosen models. It does not supply models or provider credentials; it coordinates access to them.
Technical notes+
pyproject.toml defines Python 3.10 through 3.14 support, MIT licensing, a maturin build, PyO3 bindings, three CLI entry points, and extras for proxy, thin CLI, caching, SAML, semantic routing, MLflow, gRPC, Google, Bedrock realtime, and other workloads. litellm/main.py imports provider handlers and public operations across chat, embeddings, audio, images, streaming, moderation, and fallback paths. litellm/ocr/main.py prepares provider-specific OCR requests and attempts a Rust bridge before returning validated responses. litellm/rag/main.py implements ingest and query pipelines with retrieval, optional reranking, completion, and combined cost accounting. gateway/main.py and backend/main.py reuse the proxy FastAPI app, then filter routes during lifespan startup into data-plane and management surfaces. Makefile exposes segmented unit and integration suites plus Ruff, basedpyright, format, circular-import, import-safety, and budget gates.
Observed
- License
- MIT
- Primary language
- Python, with a Rust extension built through maturin and PyO3
- Python support
- Python 3.10 through 3.14
- Install surface
- Python package with optional proxy, CLI, caching, SAML, semantic routing, MLflow, gRPC, Google, and other extras
- Interfaces
- Python library, OpenAI-compatible HTTP gateway, MCP gateway, A2A client and gateway, and three command-line entry points
- Server architecture
- FastAPI proxy application with optional route-filtered data-plane and management entry points
- Testing structure
- Segmented unit, proxy, integration, end-to-end, and Helm test targets are defined
Read from README.md, Makefile, package.json, pyproject.toml, backend/main.py, gateway/main.py, litellm/main.py, litellm/__init__.py, enterprise/__init__.py, litellm/ocr/main.py, litellm/rag/main.py, litellm/evals/main.py.
Intel on LiteLLM
Tags
Tech Stack
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.