Vibeleaderboard
← All Intel
Intel / post

Ant Ling releases six Ling-3.0 base checkpoints for continued pretraining

Source
x.com
Date
Ant Ling@AntLingAGI

🧵 We’ve open-sourced 6 Base Model checkpoints for Ling-3.0-tiny & Ling-3.0-flash, covering pre-trained, mid-trained, and WSM-merged stages. None has undergone post-training, giving researchers flexible starting points for continued pre-training, fine-tuning, and further research. Two key highlights: - We use WSM to replace LR decay with weighted checkpoint merging, making the training process better suited for continual pre-training while enabling offline exploration of different LR decay strategies. - With one shared training recipe, the community can validate strategies on tiny-base, then scale them to flash-base.

Context

A base checkpoint is a saved after on large amounts of text but before post-training, the later tuning that turns it into an assistant that follows instructions. Ant Ling open-sourced six of them, three each for Ling-3.0-tiny and Ling-3.0-flash. Per the model cards, tiny has 7.9 billion total parameters with 1.3 billion active per token, and flash has 124 billion with 5.1 billion active. For each size, Ant released a checkpoint saved at the end of large-scale pretraining (named with the suffix base-30T), one saved after mid-training, and a final base produced by applying a method called WSM to the mid-training checkpoints. None has been post-trained. The flash base model card, released under the MIT license, lists uses such as continued pretraining, , reinforcement learning and research, and advises against using a base model directly for end-user chat.

WSM, short for Warmup-Stable and Merge, addresses the end of training. Pretraining usually finishes with a learning-rate decay phase that gradually shrinks the size of each training update so the model settles. WSM skips that phase and instead builds the final model as a weighted average of checkpoints saved along the way; its paper argues that this averaging can emulate common decay schedules and reports gains over the widely used Warmup-Stable-Decay approach, including what its abstract gives as +3.5% on MATH. Because the merge happens after training, different decay-like weightings can be tried on saved checkpoints rather than by rerunning training, which is what Ant means by exploring decay strategies offline. Ant says this also makes the process better suited to continued pretraining, and the tiny base model card says both sizes share one training recipe, so researchers can test a strategy cheaply on tiny before scaling it to flash. Those are Ant's claims; neither source shows that results on tiny carry over to flash.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
  • distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
More from Ant Ling
Recommended reads
Comments

Checking sign-in…

Loading comments…