promptbase

Microsoft's promptbase repository shows how a prompting strategy can involve more than a single question. Its Medprompt method selects examples similar to the incoming question, includes model-generated reasoning examples, and asks multiple versions of a multiple-choice question with the options rearranged.

The answers are then combined by majority vote. Rearranging the choices tests whether an answer remains consistent when its position changes. This makes the repository useful for studying how to design the questioning and evaluation process around an existing model. It changes the information and repeated requests supplied to the model, without requiring a new round of model training.

“promptbase is an evolving collection of resources, best practices, and example scripts for eliciting the best performance from foundation models like GPT-4 .”

“In a recent study , we showed how the composition of several prompting strategies into a method that we refer to as Medprompt can efficiently steer generalist models like GPT-4 to achieve top performance, even when compared to models specifically finetuned for medicine.”