How we optimized Dash's relevance judge with DSPy

Dropbox used DSPy to adapt its search-relevance evaluator to OpenAI's gpt-oss-120b model. The evaluator scores how well a document answers a query. Human ratings and explanations identify disagreements, and the optimizer uses that feedback to revise the prompt. This refines the judge used in Dash's human-calibrated labeling process.

In Dropbox's reported comparison, normalized mean squared error fell from 8.83 to 4.86, about a 45% reduction. This metric penalizes larger disagreements with human scores more heavily and also penalizes malformed output. It measures better agreement and usable formatting, rather than a 45% increase in search accuracy.