distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Why it matters
Microsoft's distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition → prompt-engineering techniques and benchmarks from running LLMs at scale. Less "10 magic prompts," more the methods that actually hold up. A solid reference to pull from when a prompt is not landing.
Key quotes
“promptbase is an evolving collection of resources, best practices, and example scripts for eliciting the best performance from foundation models like GPT-4 .”
“In a recent study , we showed how the composition of several prompting strategies into a method that we refer to as Medprompt can efficiently steer generalist models like GPT-4 to achieve top performance, even when compared to models specifically finetuned for medicine.”
articlePromptimus: Improving already good LLM prompts with zero manual engineeringwww.amazon.science
articleBeyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on QualityQipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
articlePrompt EngineeringLilian Weng