pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
Why it matters
Including code in pretrainingThe first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.Full definition → improves reasoning on non-code tasks, and this quantifies how much and at what proportions. That is a direct input to data mixture decisions and to reasoning about why some base models reason better than others.
articleCommit0 Library Generation From Scratch 2024 12 02
articleLlmcrit Teaching Large Language Models To Use Criteria 2024 03 02
articleSeacrowd A Multilingual Multimodal Data Hub And Benchmark Suite For Southeast Asian Languages 2024 06 14
articleFishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 05 08