LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
If you're running dedicated LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → deployments, this explains why naive GPU-utilization-based autoscaling backfires under bursty load and gives concrete guidance on better scaling signals and cold-start budgeting to avoid latency spikes.
Key quotes
“LLM serving degrades nonlinearly: a replica at its concurrency limit doesn't get "a bit slower," it starts queueing, and TTFT can blow up from 200ms to 15s.”
Together AI
“Cold starts can take several minutes. A new replica has to be placed on a GPU node, pull tens of gigabytes of weights, load them into VRAM, and warm up. This means that you can't scale your way out of a spike because by the time it arrives it’s already too late to scale up.”
Together AI
“scale_up_window : how long the pressure must persist before adding replicas. Keep it short; the cost of a false scale-up is just a few replica-minutes, whereas the cost of a missed one is increased user-facing latency.”
Together AI
“In-flight count is a leading indicator: it rises when demand outpaces service but before latency visibly degrades.”
Together AI
“And gpu_utilization never scaled because short, bursty requests leave the GPU under its 75% bar. Both policies were reading signals that looked healthy while the system was saturated.”
articleCategories of Inference-Time Scaling for Improved LLM ReasoningSebastian Raschka, PhD
articleEnhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor ParallelismMichelle Horton
articleScaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model ParametersCharlie Snell et al.