Scaling the Memory Wall: The Rise and Roadmap of HBM
Source
Dylan Patel
Author
Dylan Patel
Date
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Memory bandwidth and capacity — not FLOPs — set the ceiling on KV-cache size, context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →, and serving cost; this explains the roadmap that will determine those limits for the next accelerator generations.
