Autoscaling Endpoints for LLM Inference

An autoscaler can leave a slow deployment unchanged if its chosen measurement stays below the trigger. Together AI illustrates this in its dedicated-inference experiment: the test client measured multi-second latency while engine-side time to first output and GPU utilization remained below their configured scaling thresholds.

Those measurements describe different parts of request handling. A GPU activity reading or an engine's first-output timing can look healthy while the complete request remains slow. The experiment supports checking an autoscaling signal against client-observed latency before relying on it to protect response times.

“LLM serving degrades nonlinearly: a replica at its concurrency limit doesn't get "a bit slower," it starts queueing, and TTFT can blow up from 200ms to 15s.”

Together AI

“Cold starts can take several minutes. A new replica has to be placed on a GPU node, pull tens of gigabytes of weights, load them into VRAM, and warm up. This means that you can't scale your way out of a spike because by the time it arrives it’s already too late to scale up.”

Together AI

“scale_up_window : how long the pressure must persist before adding replicas. Keep it short; the cost of a false scale-up is just a few replica-minutes, whereas the cost of a missed one is increased user-facing latency.”

Together AI

“In-flight count is a leading indicator: it rises when demand outpaces service but before latency visibly degrades.”

Together AI

“And gpu_utilization never scaled because short, bursty requests leave the GPU under its 75% bar. Both policies were reading signals that looked healthy while the system was saturated.”

Together AI