What does 99.9% uptime mean for inference?

An inference endpoint can accept a request while the model behind it fails to produce an answer. Together AI says it therefore measures availability at successful inference completion, rather than at the load balancer. Its reliability account also separates availability from performance: an endpoint may respond yet deliver less throughput than contracted.

The company describes continuously serving traffic from multiple facilities and retaining enough capacity to absorb a facility failure. Those details explain how a reliability promise is intended to survive an outage. A percentage alone does not identify the failures covered, the spare capacity available or whether recovery has been exercised under live traffic.

“Hitting 1M tokens per minute per GPU at 200 TPS, or sub-50ms TTFT on voice models with custom kernels, leaves limited slack. Adding reliability to a system like that is exponentially harder with each nine.”

Together AI

“a redundant system that isn’t regularly tested under real conditions is like calling a player off the bench who hasn’t practiced in six weeks.”

Together AI

“A provider renting capacity from a hyperscaler or neocloud doesn’t own their failure domains. When something breaks at the power or cooling layer, they’re filing a ticket with whoever does.”

Together AI

“The key word here is reserved. Not “we can route traffic there” but “we have room sitting idle there right now.””

Together AI

“We measure at inference completion, not at the gateway. A request that reaches the load balancer but fails at the GPU is downtime in our accounting.”

Together AI