pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
The priorities that shape a training codebase invert between a few nodes and 10,000 GPUs. Debuggability and memory headroom give way to reliability and automated ablation, and Titan is the record of that transition.
