Open Source And In House How Uber Optimizes Llm Training
- Source
- uber.com
- Author
- Uber Engineering 1
- Date
Context
Freeing GPU memory can speed training when memory capacity limits how much work fits into each step. Uber Engineering reports that moving optimizer state into CPU memory reduced GPU memory use by at least 34% at the same batch size while training Meta's Llama 2 70B. It then raised the batch size three to four times and reports two to three times the training throughput.
The smaller Llama 2 7B behaved differently: CPU offloading did not improve model flops utilization (MFU), the ratio of observed throughput to the hardware's theoretical peak, and flash helped most. Uber says the 70B run might be memory-bound and the 7B run compute-bound. Uber's experiments therefore suggest that a training optimization's value depends on whether memory or computation constrains that particular workload.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Checking sign-in…
Loading comments…