• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
• quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Serving can outcost training over a model's life, and this shows why by building the layer yourself: ONNX loading, graph construction and CPU execution in C++, with dynamic batching, memory management and quantization placed in context.

• articleThe Inference Engineering MasterclassLatent Space
• articleLarge Transformer Model Inference OptimizationLilian Weng
• articleAINews: Megakernels are so dead and so backLatent Space

Checking sign-in…

Loading comments…