Generating token-by-token leaves the big model idle between steps. Speculative decoding has a small, fast model guess the next several tokens, then the large model checks the whole guess at once — accepting the correct prefix and fixing the first mistake. Output is provably identical to what the big model would have produced alone; it just arrives faster.

It works because most tokens are easy ("the", closing brackets, obvious continuations) and a small drafter gets them right. Typical speedups run 2-3x, which is why every serious inference stack ships some variant of it.