Pushing Mixture Of Experts To The Limit Extremely Parameter Efficient Moe For Instruction Tuning 2023 09 11
Source
cohere.com
Author
Cohere
Date
Context
Instruction tuning can adjust a small set of specialist components while leaving most of a LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → unchanged. A Cohere research abstract describes a mixture-of-expertsA model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.Full definition → design built from lightweight sub-models. The authors report performance comparable to full fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → while updating less than 1% of an 11-billion-parameter model in their experiments. The finding concerns how many learned weights must change during adaptation. It does not mean the resulting system contains only that small fraction of the original model or requires proportionally less memory to run.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
postCohere releases open-weights translation model North Small Translate
articleThe Data Provenance Initiative A Large Scale Audit Of Dataset Licensing And Attribution In Ai 2023 10 25
