More accelerator memory stops helping when the data needed for active work already fits and moving that data becomes the bottleneck. SemiAnalysis develops this distinction in its inference analysis: memory capacity determines what can be held, while bandwidth determines how quickly the processor can use it.
The article argues that inactive conversation state should move to cheaper memory or storage, leaving expensive accelerator memory for current work. This requires coordinated storage, networking and scheduling. Its hardware-design implication is conditional: once active work fits with sufficient operating margin, faster data movement may be more valuable than additional local capacity.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
