How We Rebuilt Provider Routing

OpenRouter has changed how it chooses which company serves a request when several host the same model, and in its own week-long experiment requests got faster and failed less often while costing about 6% more. That choice is common: about 77% of requests on OpenRouter go to models served by more than one provider. Until now, a request that did not specify a provider was routed almost entirely by input token price: the cheaper a provider, the larger its share of traffic, and a failed request was retried at the next cheapest. The announcement by Scott Morse says this ranking let slow or failing providers keep traffic, and let a provider hold share by listing a low input price while charging more for cached or output tokens.

The new router scores each provider with what OpenRouter calls a Routing Cost, an estimate of what a usable answer really costs. It starts from the effective price, calculated from the provider's measured traffic, including its real cache hit rate and the model's usual mix of input and output tokens, which closes the low-input-price loophole. It converts waiting into cost using output speed and the time before the first token arrives, then adds the expected cost of retries: if 1 in 5 tries fails, a user pays for 1.25 tries per good answer. Rate limits, provider errors and malformed tool calls all raise the score. Traffic is then shared among providers close to the best one, with a small floor so a struggling provider can show it has recovered, and a learned cap so the leader is not overloaded.

OpenRouter measured the change with a switchback experiment rather than a standard A/B test, because the two routers share the same providers: if the new one steered traffic away from a struggling provider, the old one's requests there would speed up and blur the comparison. It alternated matched 3-hour blocks between the routers for one week, discarded the first hour after each switch, and split more than 24 million requests across a sample of open-source models. Median time to first token fell from 1,007 to 837 milliseconds, median output speed rose from 62.4 to 72.6 tokens per second, the share of requests that hit a rate limit fell from 7.1% to 3.8%, and the share that needed a retry fell from 10.4% to 6.9%. Cost per thousand requests rose from $1.25 to $1.32.

Developers do not need to change code, and those who want the old behavior can keep it. Setting provider.sort to price, appending :floor to the model name or choosing price as the account default restores price ordering, and an explicit provider order bypasses the new router entirely, as the provider routing guide describes. An ongoing conversation stays with its first provider so its prompt cache remains useful. The figures are OpenRouter's own measurement of its own system over one week, and the company says providers have not yet had time to adjust to the new scoring, so whether the speed gain and the 6% premium hold as they adapt is still open.