Clip / Entertainment

Origin of the memory wall

From Why LLM Inference Is Memory-Bound (Julia Turc) · ≈0:30

Frames the fundamental hardware trend behind why inference is memory-bound rather than compute-bound.

What’s in it

Frames the fundamental hardware trend behind why inference is memory-bound rather than compute-bound.

Clip transcript

This problem can be traced way back. For decades, Moore's Law kept doubling transistor density and implicitly raw compute. Memory, however, has been improving a lot more slowly. This imbalance became known as the memory wall. Around 2019, when AI model sizes exploded, the memory wall became a bottleneck for LLM inference. In this video, we'll see why that's the case and what the