• context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
• inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Splitting prefill and decode onto different silicon changes the economics of long-context inference — the phase that dominates cost when agents feed large contexts on every turn.

• articleScaling the Memory Wall: The Rise and Roadmap of HBM
• articleHuawei Ascend Production Ramp: Die Banks, TSMC Continued Production, HBM is The Bottleneck
• articleH100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time

• articleInside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AIEduardo Alvarez
• articleVera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture AnalysisAlec Ibarra
• postRubin Ultra's HBM lands at 192GB, a fifth of its previewSemiAnalysis

Checking sign-in…

Loading comments…