So the weights need to be stored somewhere. GPUs typically come with their own on-chip memory. This is static RAM or SRAM. It's fast but expensive to manufacture and takes a lot of physical space, which is why there's only so much of it. NVIDIA's H100 GPU has a bit over 50 megabytes of SRAM. That's not nearly enough for state-of-the-art LLMs. Open weight models go up to hundreds of gigabytes of storage and proprietary ones must be even larger. To support this demand, GPUs come with yet another level of memory placed near the processor, but not on the same chip. It's called high bandwidth memory or HBM. This is typically what people refer to when they say "GPU memory". Unlike SRAM, it uses a stacked die architecture that trades some speed for much greater capacity. An H100 comes with 80 gigabytes, much more suitable for storing the model weights.