pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
Why it matters
A candid account of building a streamable pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → corpus end to end — source mixing, caption philosophy, latent and shard formats, resolution bucketing, dedup — from a team that shipped the model on top of it.
videoTraining Krea 2: What matters in generative model training — Sangwu Lee, Krea.aiAI Engineer
videoThe Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolsideAI Engineer
videoData and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke LabsAI Engineer