A prompt change can alter an assistant's answers even when the surrounding software still builds successfully. OpenRouter's evaluation-gate tutorial addresses this by running fixed input-and-acceptance tests when relevant files change, then blocking a merge if results fall below a threshold. Repeated runs of the unchanged system establish how much scores vary before that threshold is chosen.
The tests and grading rules also need review, because weakening them can conceal a regression. A separate dataset guide explains how to build reviewed examples from real usage.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
