Vibeleaderboard
← All Intel
Intel / article

Introduction to GPU Inference, a short course with Interactive diagrams

Source
athrael.net
Author
athrael-soju
Date

Context

Introduction to GPU is a free, voice-narrated course on Athrael.net that teaches how AI models are served to many users at once, starting from zero (course page). Its author, who signs as Athos, says in his Hacker News post that he spent most of this year working on GPU inference and that the course needs no signup or fee. The lessons run 6 to 11 minutes, each with diagrams the reader can operate. They begin with what inference is, a trained model producing a reply one (a small piece of text) at a time, and work toward how a large service is run.

The middle lessons explain the measurements that decide whether a service feels fast. Batching computes the next token for several requests in one step, which raises the GPU's total output. Continuous batching lets a waiting request take a slot as soon as any reply finishes, instead of waiting for the whole batch. The course separates time to first token, the wait before a reply starts, from time per output token, the gap between tokens after that: at 20 milliseconds per token, text arrives at 50 tokens per second. Goodput counts only the requests that met a latency target, so it falls when a server is overloaded even if the GPU is producing plenty of tokens.

The most advanced lesson so far covers serving architectures. Prefill processes the whole prompt in one pass; decode then generates the reply token by token. When every copy of the model does both, a long new prompt slows the replies already in progress. Disaggregated serving puts prefill and decode on separate copies and passes the , the intermediate results computed from the prompt that decoding reuses, from one to the other. The lesson also explains adaptive allocation, where copies of the model switch between the prefill and decode groups while the service runs, as demand shifts.

The course is unfinished. Two later lessons, on serving frameworks and on running a first live inference server, are marked coming soon, and the consumer-GPU lesson on adaptive disaggregated serving that the author promises on Hacker News is not yet posted. What exists is a conceptual introduction rather than a guide to any particular serving software. For how these same measurements become practical problems when a company buys inference from a provider, the Pragmatic Engineer's guide to inference engineering treats long waits for the first token as a cost of sharing machines with other customers.

Terms in this piece · Glossary
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Read the source athrael.net
Recommended reads
Comments

Checking sign-in…

Loading comments…