• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.

The gap between a model's weight size and what it actually needs on the card is where most local deployments fail, and this shows the arithmetic.

• articleSmaller, faster, safer: running Kimi and GLM at scaleCloudflare
• clipQuantization degradation benchmarksBijan Bowen
• articleEfficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon et al.

Checking sign-in…

Loading comments…