• pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
• distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
• attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.

Architecture ideas can be tested by editing a pretrained model instead of pretraining from scratch. Activation distillation combined with short fine-tuning can swap attention for convolution or linear attention while holding quality.

• articleConvolutional Multi Hybrids For Edge Devices
• articleHow Effectively Does A Model Use Its Memory
• articleLfm 1b Math Can Small Models Be Concise Reasoners
• articleAutomated Architecture Synthesis Via Targeted Evolution

• videoVision Transformer, Diffusion Transformer, and Multimodal Diffusion Transformer ExplainedJulia Turc
• videoFrom Transformers to Diffusion LLMs: Understanding LLaDAJulia Turc
• articleDual-Flow Transformers: Decoupling the Primary Prefill Path from Additional Decode ComputationLiming Liu, Mingze Wang, Tuo Zhao

Checking sign-in…

Loading comments…