Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

In synchronous reinforcement learning, a training update waits for the batch's generated responses to finish, so a few long responses can hold up everyone else. Together AI's distribution-aware speculative decoding uses recent responses as a searchable history of likely continuations. The current model verifies those proposals, while scheduling spreads long requests across GPUs and gives them more speculative work. The method description explains this combination.

The history-based drafter requires no gradient training. In the reported experiments, the combined system reduced response-generation time by more than 50% for mathematical reasoning and about 25% for code generation while preserving measured reward quality. These savings apply to the rollout phase, the part of training that generates responses, rather than the entire training run.

“70% of total training time is consumed by the rollout phase — exceeding the cost of backpropagation and parameter updates combined.”

“A static drafter trained on an earlier checkpoint quickly becomes misaligned with the current policy.”

“The design is simple enough to describe in a few paragraphs. The results are what validate it.”

“DAS is distribution-preserving — identical outputs to standard decoding, identical training curves.”