• KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
• distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Two models with identical cache sizes can differ sharply in memory utilization; ESS measures the gap and predicts which models distill badly and which handle recall-heavy context well.

• articleConvolutional Multi Hybrids For Edge Devices
• articleLfm 1b Math Can Small Models Be Concise Reasoners
• articleAutomated Architecture Synthesis Via Targeted Evolution
• articleExploring Diffusion Transformer Designs Via Grafting

• articleHow Much Memory Does Your Agent Actually Need?huggingface.co
• articleEfficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon et al.
• articleCan I Run This Model?Aadi1277

Checking sign-in…

Loading comments…