Large Transformer Model Inference Optimization
lilianweng.github.io- Category
- Other
- Type
- ARTICLE
- Builder
- @lilianweng
- Added
- Jul 21, 2026
About
[Updated on 2023-01-24: add a small section on Distillation .] Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale. Why is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factor
Why it made the leaderboard
A rigorous survey of how to cut transformer inference cost in time and memory — quantization, pruning, sparsity, and distillation — giving engineers a practical map for deploying large models affordably at scale.
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.