Hardware Performance Trend

This October 2024 data insight from Epoch AI, by Robi Rahman and David Owen, measures how much faster AI chips get when they use tensor cores and smaller number formats instead of ordinary 32-bit floating point (FP32) math (epoch.ai). Tensor cores are specialized units for the matrix operations at the heart of neural networks, and lower-precision formats such as TF32, FP16 and the 8-bit integer format INT8 store each number in fewer bits, so more operations fit through the same hardware.

Across the trends in Epoch's hardware dataset, TF32, tensor-FP16 and tensor-INT8 deliver around 6x, 11x and 13x the performance of non-tensor FP32 on average. Individual chips can go far beyond that: Epoch notes the NVIDIA H100 is 59x faster in INT8 than in FP32. Epoch says these format gains account for about half of the overall improvement in the performance trend since they were introduced, and that training in lower precision, especially tensor-FP16, has become common as developers take advantage of the boost.

The multipliers come from separate log-linear regressions of performance against release date for each format, compared over the span where the shorter data series overlap, typically from around 2016. They describe aggregate performance trends across many accelerators, not a guaranteed speedup for any given model or chip; Epoch notes individual chips vary significantly in how much they benefit. Epoch flags that the result assumes its broad set of ML accelerators is representative even though a few leading chips are used more often, and that the per-format trends stay roughly parallel.