Index / tool

Switchyard

Category: Developer Tools
Rank: No. 930Tools index
Pricing: Open Source
Type: TOOL
Use case: Models: Train & Run
Interfaces: CLI · API
Builder: nvidia-nemo
GitHub: 3.5k stars
Latest release: v0.4.0a1
Date: Indexed Aug 15, 2026

About

Switchyard is an NVIDIA NeMo project that sits between an LLM application and its model providers, routing each request and translating between the OpenAI and Anthropic API formats so a client written for one can talk to the other. That translation is what lets coding agents such as Claude Code or Codex CLI run against open-source models. It offers typed, composable routing algorithms including random, LLM-classifier, stage, and escalation routers, records operational metrics for benchmarking, and can be used as a CLI launcher, a standalone server, or an embedded Rust library.

What it does

A model picker for agents. You declare a pool, usually a cheap model and a stronger one, and a policy decides per request which should answer. Policies can look at the agent's recent tool results and errors, ask a judge model whether the cheap one can cope, or blend both. Every incoming request is decoded into neutral types first, so a Claude Code session can be served by a Chat Completions backend and still get replies in the shape it expects. It runs as a hosted OpenRouter option, a gateway plugin, a local proxy, or a library inside your own harness.

Why it's ranked here

NVIDIA ships it under Apache 2.0 with a real engineering footprint: a twelve-crate Rust workspace, a Python package built with maturin covering Python 3.10 to 3.14, a dry-run config check, and published benchmark tooling for comparing routing against a single-model baseline. The project is also unusually candid. It labels each component Beta, Alpha or Demo, calls itself pre-1.0, and keeps a known-issues page listing concrete bugs. That combination of serious backing and honest maturity labels is why it belongs on the list, with the caveat that most of it is still experimental.

What's good

Credentials stay out of config: the routing file names an environment variable and the server reads the key at startup. A dry run validates schema, key lookup, target references and route construction without opening a socket. The embedded library never calls a model itself; it hands every judge call and final call back to you, so it fits a host that already owns retries and auth. The server forwards rate-limit and trace headers from upstream, caps buffered request bodies at 32 MB, and the docs warn plainly that a cheaper call does not guarantee a cheaper finished task.

Tradeoffs

The standalone server is labelled Demo, for single-user machines and explicitly not production. The routing library is Beta and the client and runner are Alpha, with API changes promised before 1.0. Validation covers Ubuntu 24.04 on x86_64 only. The LiteLLM plugin cannot serve classifier or escalation strategies, and NeMo Relay 0.8 to 0.9.0 can drop upstream error details when the plugin is on. Known issues include buffered upstream work continuing after a client disconnects, which can still cost money, and routing-tier attribution missing from stats for some fallback decisions. The upstream wire format must be declared; nothing is probed automatically.

How to use it well

Start with the OpenRouter-hosted option or the local proxy, pick the Auto preset, and run your agent on a real task set against your current single model before trusting any savings. Use the dry-run check whenever you edit the routing file. If you run your own gateway, embed the library rather than the demo server, since it leaves transport, retries and credentials with you. This is for teams paying for a top-tier model on agent work that is often easy. It does not host models, manage keys across users, or give you a production-grade proxy today.

Technical notes+

Cargo.toml defines a resolver-3 workspace of twelve members (libsy, libsy-llm-client, protocol, switchyard-translation, switchyard-runner, switchyard-server, prefill-router, switchyard-py and others) at edition 2024. pyproject.toml publishes nemo-switchyard via maturin with zero runtime dependencies and mypy strict. crates/libsy/src/lib.rs exports the algorithm set: StageRouter, LlmTaskClassifier, CompositeRouter, PlanExecute, SubagentRouter, AdvisorGate, Random, Passthrough, plus a Step/drive stream API where Step::CallModel returns judge calls to the host. crates/switchyard-server/src/lib.rs is axum-based, sets DEFAULT_MAX_REQUEST_BODY_BYTES to 32 MiB, a 30 second graceful shutdown, optional rustls TLS, a Prometheus registry and an allowlist of forwarded upstream headers. crates/prefill-router/src/lib.rs defines a PrefillForward trait for learned per-model probability routing.

Observed

License: Apache 2.0
Language: Rust workspace with Python bindings built via maturin
Python support: Python 3.10 through 3.14
Interfaces: Standalone HTTP proxy, embeddable Rust library, Python package, gateway plugins (OpenRouter, LiteLLM, NeMo Relay)
Client API formats: OpenAI Chat Completions, OpenAI Responses, Anthropic Messages
Install surface: cargo install switchyard-server; pip package nemo-switchyard
Platform validation: Standalone server release-validated on Ubuntu 24.04, Linux x86_64 only
Component stability: Library Beta, client and runner Alpha, server Demo (not for production)
Configuration: TOML file; API keys referenced by environment variable name

Read from README.md, Cargo.toml, pyproject.toml, LICENSE, docs/architecture.md, docs/core_concepts.md, docs/getting_started.md, docs/known_issues.md, crates/switchyard-server/src/main.rs, crates/switchyard-server/src/lib.rs, crates/libsy/src/lib.rs, crates/prefill-router/src/lib.rs.

Intel on Switchyard

Tech Stack

PythonRustDocker

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.