Vibeleaderboard
← All Intel
Intel / article

Can I Run This Model?

Source
nerdylive123.github.io
Author
Aadi1277
Date

Context

Can I Run This Model? explains why a 's stored weights are only part of its GPU-memory requirement. Its calculator adds memory for the cache, which retains information used while processing a sequence, and runtime overhead.

In the displayed formula, cache memory grows with and the number of simultaneous sequences. A configuration that fits a short, single conversation can therefore require more memory when either setting increases. The page exposes its arithmetic so users can see which component changes the estimate. These are planning estimates, rather than benchmark results proving that a particular deployment will fit or achieve a given speed.

Terms in this piece · Glossary
  • Large language model — A model trained on large amounts of data to process and generate language.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Read the source nerdylive123.github.io
Recommended reads
Comments

Checking sign-in…

Loading comments…