AI Model Co-Design: Hardware-Friendly LLM Design

A model can generate plenty of text across a data center while still feeling slow to each person using it. NVIDIA's hardware-design guide separates aggregate throughput from responsiveness, which depends on waiting for the first output and subsequent text. A useful design must meet its response-time target while preserving accuracy. The guide explains this tradeoff.

The model's internal dimensions help determine that balance. GPU computation involves moving data as well as performing arithmetic; poorly chosen matrix dimensions can leave the processor waiting on memory. Designing those dimensions for the hardware can improve utilization. The guide therefore treats deployment requirements as inputs to model design, rather than something to address only after training. Its linear-layer analysis provides the mechanism.