GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta reports doubling the training efficiency of GEM, its ads recommendation foundation model, to 20–25% model FLOPs utilization. That measures useful model arithmetic relative to available compute capacity across training. The engineering account attributes the improvement to coordinated changes in computation and distributed execution.

One concrete obstacle is uneven user-history length: padding every example to the longest history spends computation on empty positions. Specialized kernels operate on variable-length inputs directly. Other changes address communication and workload imbalance across GPUs. The reported doubling concerns training efficiency, while the total training computation expanded fourfold over the same twelve-month period.