Intel / article

Aurora

Source: together.ai
Author: Together AI
Date: Published Mar 31, 2026

Context

Aurora updates the small model that proposes text for a larger model to verify. That division of labor comes from speculative decoding, whose foundational paper describes checking several proposed together. Aurora addresses what happens when changing traffic makes those proposals less useful.

Together AI's Aurora framework sends accepted and rejected proposals to a separate training process, which periodically supplies updated draft-model weights while serving continues. In simulated traffic spanning five domains, the authors report an additional 1.25-fold speedup over a trained but static drafter. Recovery after abrupt domain shifts took roughly 10,000 requests. The adaptation improves the drafting process; it does not retrain the target model.

Terms in this piece · Glossary

speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Key quotes

“The verifier moves, but the drafter lags. Production target models change — for quality, safety, cost, or hardware migration. The speculator often updates much more slowly, so it becomes stale and speculative performance degrades over time.”
Together AI

“Offline distillation pipelines are expensive. Activation collection and replay pipelines for drafter training can be extremely costly to store and operate at scale. At production scale, the storage footprint can reach petabyte-level magnitude, with high cost in memory, bandwidth, and operational complexity.”
Together AI

“Acceptance rate is not the same as real speedup. Offline training can optimize acceptance in a lab setting, but production speedup depends on the actual serving stack: kernels, numeric precision (FP8/FP4), batching, scheduling, and hardware behavior. The best draft model offline may not be the best model online.”
Together AI

“The system recovers acceptance length within approximately 10,000 requests after each shift, demonstrating robust online adaptation.”
Together AI

“The mixed traffic results are particularly striking: online training from scratch can exceed the performance of a carefully pretrained speculator. The acceptance length reaches 3.08 (surpassing both the static baseline at 2.63 and the pretrained-then-finetuned baseline at 2.99), with throughput stabilizing at 302.3 tokens/s.”
Together AI

Read the source www.together.ai