Automated Genai Driven Search Quality Evaluation

LinkedIn's automated evaluator checks the suggestions that appear while someone types into search. Engineers define what counts as a good suggestion for each result type, then ask a language model to grade suggestions against those guidelines using sampled queries and user context. Comparing scores across experiments gives the team a repeatable way to assess changes. LinkedIn reports completing evaluations in hours rather than the days or weeks manual review could require. Human judgment remains part of maintaining the evaluator: engineers used disagreements with human assessments to revise its prompts.