• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

Long context is expensive because the KV cache grows with every token. SnapKV predicts which tokens matter from the prompt's own attention pattern and drops the rest, cutting memory and latency without retraining the model.

• articleCommit0 Library Generation From Scratch 2024 12 02
• articleLlmcrit Teaching Large Language Models To Use Criteria 2024 03 02
• articleSeacrowd A Multilingual Multimodal Data Hub And Benchmark Suite For Southeast Asian Languages 2024 06 14
• articleFishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 05 08

• clipA continually-adapting relevance scorer, and KV-cache-aware injectionAI Engineer
• articleUnderstanding and Coding the KV Cache in LLMs from ScratchSebastian Raschka, PhD
• articleRecent Developments in LLM Architectures: KV Sharing, mHC, and Compressed AttentionSebastian Raschka, PhD

Checking sign-in…

Loading comments…