Model Fallbacks with LangSmith LLM Gateway

LangChain's post promotes model fallbacks in LangSmith LLM Gateway, which the company's announcement put into public beta for its Plus and Enterprise plans on July 30, 2026. A gateway is a single address that agents send their model requests to. It forwards each request to a provider such as OpenAI, Anthropic or Fireworks and applies shared rules on the way, including spend caps, rate limits and fallbacks. LangChain's argument is that without one, engineers write those controls into each agent for each provider, and the announcement's example is a provider outage that leaves a customer service agent returning errors. Getting started, as the announcement describes it, mostly means pointing existing code at the gateway's address and adding provider keys.

A fallback chain is an ordered list of models to try. The post's screenshot shows one that starts with a Claude Sonnet 5 configuration and falls through to Claude Opus 5 and then Claude Opus 4.8. In the documentation at launch, the gateway moved to the next model only when a provider returned an HTTP status code chosen as a trigger, such as 429 for a rate limit or 500 and 503 for provider errors. The feature has changed since. The current fallback documentation describes a primary model with one to five backups, default triggers that include 408, 429, 500 and 503, and two further reasons to move on: no output within 120 seconds by default, or a failed connection.

The current documentation also marks where fallback stops. Once a streaming response has started sending output, the gateway commits to that model and does not switch even if the stream later fails. A 429 produced by the gateway's own rate-limit policy blocks the request without trying the chain, since only a provider's 429 counts. Both versions of the documentation say spend policies count each provider attempt separately, so a request that falls through several models records one call for every model it tried.