Building A Golden Eval Dataset From Production Traffic

A useful application test set pairs real user requests with reviewed criteria for acceptable answers. OpenRouter's golden-dataset guide recommends sampling production traffic, removing sensitive information and duplicates, and retaining coverage of consequential failures. Reviewers then define the facts, actions or formatting each response must satisfy. Running the existing system first helps expose ambiguous cases and faulty grading rules before they become false alarms. Keeping the examples, rubric and baseline together under version control makes later score changes easier to interpret.