Scaling Llm Based Ranking Systems With Sglang At Linkedin

LinkedIn's engineers reworked the open-source SGLang serving system for ranking, a job in which a language model reads a search query and a candidate and returns a relevance score instead of writing text. In their February 2026 post, the changes skip the text-generation steps that ranking does not use, process requests in batches, and compute the shared query once rather than for every candidate. LinkedIn reports that the combined optimizations raised throughput for a 375-million-parameter ranker on short inputs (a 50-token query and 150-token items) from 750 to 2,200 items per second per H100 GPU, while keeping 99th-percentile latency at or below 500 milliseconds. The figure is cumulative and applies to that model and setup.

The post describes two ways of reusing the query computation. Multi-Item Scoring, covered in LinkedIn's earlier article, concatenates all candidates into one sequence and masks attention between them. In-batch prefix caching keeps ordinary separate inputs and reuses the shared query's computation within a single pass. LinkedIn reports similar throughput for the two on a smaller test model (about 2,100 versus 2,200 items per second), so the difference is where the optimization sits in the software.