Accelerating LLM Inference via Vector Index Based Output Embeddings
Source
arxiv.org
Author
Martin Loretz, Sepp Hochreiter
Date
Context
A small LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → can spend substantial effort scoring its entire vocabulary for each next tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. This inference study replaces that exhaustive final step with an index that searches for a smaller set of promising candidates. The index is built from the model's existing output vectors, and the authors say it preserves the model's internal representations.
For Google DeepMind's Gemma 3 270M, the authors report up to 82% higher end-to-end generation throughput with one sequence at a time on a CPU. Their tests used 32-bit precision, and beyond a batch size of 32 the search overhead made the method fall behind for some models. The gain therefore applies to a specific small-batch setting.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
articleRecipes for Steering and Scaling LLMs via SamplingJiajun He, Zongyu Guo, Jos\'e Miguel Hern\'andez-Lobato, Yuanqi Du
articleGiga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text EmbeddingsEgor Kolodin, Egor Krasnoperov, Evgeniy Kosarev, Fyodor Minkin
articleX-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative DecodingJaeduk Lee, Wan Choi