← All IntelIntel / article
Turbocharging Linkedins Recommendation Systems With Sglang
- Source
- linkedin.com
- Author
- LinkedIn Engineering
- Date
Context
LinkedIn's engineering team reduced repeated work when an AI model ranks recommendations for one person. Its modified SGLang serving system processes the person's profile and history alongside several candidate items in one request. Each item can use that shared background, while an restriction prevents it from reading the other candidates.
For a request ranking 50 items, LinkedIn reports 69% lower latency than scoring items separately, even when the comparison system reused cached background information. The result concerns how quickly the ranking computation runs; it gives engineers a specific way to avoid repeatedly processing the same user .
Terms in this piece · Glossary
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Read the source www.linkedin.com
More from LinkedIn Engineering
Recommended reads
- articleScaling Llm Based Ranking Systems With Sglang At LinkedinLinkedIn Engineering
- articleAccelerating Llm Inference With Speculative Decoding Lessons From Linkedins Hiring AssistantLinkedIn Engineering
- articleRebuilding Linkedins Follows Recommendations With Llm Based Semantic Retrieval And RankingLinkedIn Engineering
Comments
Checking sign-in…
Loading comments…