Intel / article

Intra-Prompt Parallel Decoding for Common-Context Question Answering

Source: arxiv.org
Author: Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Antonios Valkanas, Yilun Zhu, Shervin Malmasi
Date: Published Sep 9, 2026

Context

When several questions refer to the same material, answering each through a separate prompt repeats work. Intra-Prompt Parallel Decoding puts those questions into one prompt and generates the next piece of each answer in the same processing step. It adjusts position information and which parts of the prompt each answer can attend to, sharing memory and computation without retraining the model. Research abstract.

The authors report up to seven times the effective throughput of standard decoding without reduced answer quality in their experiments. The opportunity is specific to groups of questions sharing , where the system can reuse that common material while producing separate answers.

Terms in this piece · Glossary

context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Key quotes

“We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt.”

“Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings.”

Read the source arxiv.org