A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
Source
arxiv.org
Author
William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, Juncheng Yang
Date
Context
A short sample of AI requests can miss how demand changes across models, users and time. This workload study analyzes a full year of production activity from a model-serving service, covering both popular models and less frequently used ones.
The researchers examine aggregate traffic alongside changes over time and differences among models and users. That scope is intended to support more realistic studies of serving infrastructure than sampled or synthetic workloads alone. The abstract promises a release of the trace, but does not establish that it is already downloadable or report a quantified improvement to caching or load balancing.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
articleGardenBench: A lightweight, daily evaluation of LLM capabilitiesBenedict Cummins-Mburu, Mariana Garcia Mejia, Oscar Heath, Sabrina Kreyzerman, Ellie Murray, Arusan Surendiran, Rohan Alexander
articleRecipes for Steering and Scaling LLMs via SamplingJiajun He, Zongyu Guo, Jos\'e Miguel Hern\'andez-Lobato, Yuanqi Du
articleAccelerating LLM Inference via Vector Index Based Output EmbeddingsMartin Loretz, Sepp Hochreiter