context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
Why it matters
This decoding method answers several questions sharing the same context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → within a single prompt and inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → pass, cutting the attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → memory bottleneck without changing output quality versus separate prompts.
articleHow Do Prompt Variations Affect Energy Consumption in On-Device LLMs?Wei Hu, Xiaolong Tu, Dawei Chen, Yitao Chen, Kyungtae Han, Haoxin Wang
articleCache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM servingTogether AI
articleXHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question AnsweringIman Barati, Arash Ghafouri, Behrouz Minaei-Bidgoli