Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Osprey aims to make the small models used to accelerate text generation more reusable. It builds on speculative decoding: a small model drafts several pieces of text, and a larger model checks them together. The original method gains speed by letting the larger model verify proposed text in parallel.

Osprey starts with an already pretrained small language model, removes layers to make it faster, then retrains the shortened model before adapting it to a particular larger model. The shared preparation can be reused, although each deployed drafter still needs target-specific adaptation. The researchers report improved acceptance of drafted text across three target models, with the largest gains on unfamiliar task distributions and multilingual data.

“This is a striking inversion of modern LLM development, where target models are valued precisely for the broad generalization they acquire through large-scale pretraining.”

“Osprey addresses both by pruning to a shallow backbone, restoring its language-modeling capability with target-agnostic next-token pretraining, and adapting it to each target through vocabulary alignment, zero-initialized QKV expansion, and distillation from the target model's output distribution.”