Pre-anneal Nano/Mini checkpoints are live. Same strong math + code foundation, with more room to fine-tune and adapt. Excited to see what you build. https://t.co/VNtXnxIUME
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Why it matters
These checkpoints sit before the learning-rate anneal, which the team says makes them easier to continue pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → on than the released bases. That matters if you are adapting a small mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → to a new domain rather than fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → at the margin.
