GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Source
engineering.fb.com
Date
Why it matters
LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → training recipes — kernels, parallelism, low-precision schemes — don't transfer cleanly to recommendation workloads. Meta reports doubling end-to-end efficiency to 20-25% MFU via custom jagged-attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → kernels, MXFP8, and topology-aware 5D parallelism.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
