This recent paper called FOCUS claims that 90% of computations within a refinement step are wasted. The model can only confidently predict about 10% of the tokens. For instance, the model was confident that Harry must be followed by Potter. But the tokens surrounding the conjunction "and" were very ambiguous because a lot of tokens would've made sense here. The FOCUS model attempts to find these high confidence positions ahead of time and only decode those tokens. It's a bit of a greedy strategy that minimizes risk at each step by focusing on low ambiguity positions first, and thus reducing wasted computations. Another source of waste is generating a full context window of thousands of tokens even for prompts that call for a short response. This paper dynamically adjusts the context window across refinements. On the first iteration, they start with a small context window of 64 tokens. Once tokens are decoded, they analyze the probabilities associated with the end of sequence token everywhere in the window. If they're low across the board, they extend the window size for the next iteration. When the end of sequence token starts getting assigned reasonably high probabilities, they lock the response length and continue refining it during the subsequent iterations.