But what if we could loosen that constraint? This is the intuition behind the speculative decoding. The decoding steps are delegated to a smaller draft model, which performs forward passes a lot faster. The bigger LLM is repurposed as a verifier for the draft. Given that the draft tokens are readily available, it can now make multiple predictions in a single forward pass similar to the prefill stage we discussed earlier. The longest common prefix between the draft and verifier outputs is committed to the response and the disagreeing tokens are discarded. If we do some quick accounting for this particular example, speculative decoding costs us three light forward passes and a heavy one with an output of two tokens. Regular decoding would've cost us three heavy forward passes with a return of three tokens. So the math works out in favor of speculative decoding only when the draft model is chosen strategically. Too large and it loses its speed advantage over the verifier. Too small and its predictions are always discarded leading to more wasted computations. So it's a tricky balance.