- Category
- Developer Tools
- Rank
- No. 1889Tools index
Previous survey · No. 1805 ·
- Type
- TOOL
- Use case
- Models: Train & Run
- GitHub
- 7 stars
- Latest release
- v0.1.1
- Date
About
Relay is a self-hostable AI gateway that classifies each incoming request to determine which model capabilities it needs, then routes it across configured providers while tracking each provider's rate limits and available capacity. It uses a queue-first approach, pacing or briefly delaying requests to stay within a preferred provider's limits rather than immediately falling back to a worse model, which the author built after hitting inconsistent rate limits while chaining several providers' free tiers.
At a glance
AnchorShell Relay routes model requests through one OpenAI-compatible endpoint with queues and ordered fallbacks. It is worth evaluating if your application hits provider limits and you want to control when it waits and when it changes models.
Good fit
Developers who use several providers and can test routing against their latency and quality requirements.
Consider something else when
Use one direct provider when it meets your needs and you do not want a proxy handling prompts and credentials.
Start here
Configure a scratch group with two test routes. Simulate an unavailable primary and inspect whether the request waits, falls back or returns an error as intended.
Cost and requirements
The project offers a community edition and a hosted service with different routing features. Provider usage and credentials are separate.
Limits to know
A fallback can change response quality, behavior and data destination. Review which request fields appear in logs.
Sources checked 2026-11-05. Source-based assessment.
Why it made the leaderboard
Relay classifies and routes requests across LLM providers using its own trained classification models, then queues and paces calls against rate limits instead of falling back immediately, useful for anyone stitching together multiple providers.
Intel on Relay
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.