How to Size GPUs for AI Inference and TCO Without Overspending
Source
Elizabeth Goodman
Author
Elizabeth Goodman
Date
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
Why it matters
Turns GPU capacity planning into arithmetic you can do before signing a contract. FP8 post-training quantizationShrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.Full definition → cut Llama-3.1-8B weight memory by 43.5% with no retraining.
