speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
If you're doing RL post-training, rollout generation dominates wall-clock time; distribution-aware speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → adapts the draft/verify process to the evolving policy distribution to cut rollout time without hurting reward quality, unlike static speculative decoding tuned for fixed inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → workloads.
Key quotes
“In RL training, the model must generate complete responses to every prompt in a batch before the next training step can begin. The slowest generation determines total step time — a textbook long-tail problem.”
Together AI
“70% of total training time is consumed by the rollout phase — exceeding the cost of backpropagation and parameter updates combined.”
Together AI
“DAS constructs a suffix tree from a sliding window of recent trajectories. During decoding, it finds the prefix match between the current context and the indexed history. Candidate next tokens are then scored by their frequency in the matched subtree, and the highest-scoring token is selected as the speculative draft.”
Together AI
“Within each GPU, requests are dynamically partitioned into Long, Medium, and Short categories based on historical rollout statistics. Long requests receive an aggressive speculative decoding budget, medium requests use a moderate budget, and short requests skip speculation entirely”
Together AI
“DSR-sub dataset (1,209 examples). DAS achieves over 50% rollout time reduction while matching the baseline reward curve exactly.”
