Accelerating Llm Inference With Speculative Decoding Lessons From Linkedins Hiring Assistant

LinkedIn's Hiring Assistant, an AI agent for recruiters, speeds up structured recruiting analyses by proposing text sequences already present in its input. The main model checks several proposed text units together and accepts suitable continuations. Repeated job titles, skills and response structures make this prompt-lookup approach useful without serving a separate drafting model.

In its November 2025 account, LinkedIn reports an average 66% reduction in its 90th-percentile end-to-end latency, with no quality degradation in internal evaluations. This applies the principle behind earlier speculative-decoding research: parallel verification can preserve the target model's output distribution. Here, recurring language in the workload supplies inexpensive proposals.