DSPy course: stop hand-tuning prompts, optimize them against evals
- Source
- x.com
- Date
New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with @databricks and taught by @ChenMoneyQ, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here:
Context
DSPy turns prompt improvement into a process guided by examples. The course announcement describes automatically tuning prompts, including the examples placed inside them, instead of repeatedly editing their wording by hand. The publisher description connects that optimization to modular programs and MLflow traces, which expose what individual steps did. The developer still defines the task and provides evidence of desired behavior; the optimizer searches for prompts that perform better against that evidence.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleHow we used DSPy to turn AI evaluations into better responses in Dash chatIlya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon
videoThe Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest & Isaac MillerAI Engineer
articleHow we optimized Dash's relevance judge with DSPyIlya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon
Checking sign-in…
Loading comments…