SGLang, software for serving language models, adds beam search in version 0.5.19. A request can explore several candidate sequences and return the highest-ranked ones, giving an application alternatives to a single generated continuation.
The implementation makes this a per-request choice: beam-search requests can share batches and memory pools with ordinary requests without a separate server switch. The release excludes combinations with speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition → and hierarchical caching, among other modes, so adopting it depends on the serving configuration.
Terms in this piece · Glossary
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
