Large Transformer Model Inference Optimization
- Source
- lilianweng.github.io
- Author
- Lilian Weng
- Date
Context
Storing a model's numbers at lower precision can reduce its memory needs, but faster responses depend on the hardware and implementation too. Lilian Weng's inference optimization survey explains this distinction through , which represents model values using fewer bits.
The numerical conversion also needs care. Some computations contain unusually large values that simple low-precision schemes handle poorly, reducing model quality. Methods can preserve higher precision for sensitive values or adjust how numbers are represented. A useful optimization must therefore account for the model's numerical behavior and the processor's ability to execute the resulting operations efficiently.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
- transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
“The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.”
Checking sign-in…
Loading comments…