inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
L40S GPU hours drop to $1.25, and the provider's own demand data shows cheaper A10s dominate because they handle mid-sized generative workloads like Mistral Nemo and Stable Diffusion well enough.
articleUltra-High Interactivity on NVIDIA GPUs? - TileRT InferenceXBryan Shan
articleH100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over TimeDylan Patel
podcastDylan Patel — Deep dive on the 3 big bottlenecks to scaling AI computeDwarkesh Patel