speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
The result offers measured numbers for AI-written systems work: 20% lower serving cost from generated GPU kernels and 15% or more from speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition →, both of which are direct inputs into where inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → prices head next.
