How @Similarweb evaluates a Deep Research agent when there's no single right ans

Grading a long research report written by an AI agent cannot work like grading a quiz, because many different reports can be correct. In a post on LangChain's blog, Liora Korni, a senior AI engineer at Similarweb, describes how the team tests changes to Data Studio, an agent that answers plain-language questions about website and app traffic by calling Similarweb's data tools and writing up the results. For a quick factual question, the team compares the agent's answer with a stored reference answer. For its Deep Research mode, which produces long reports, a reference report would only reward similarity, since one good report might focus on traffic sources and another on competitive position.

So the team scores those reports several ways at once. Plain code checks whether the agent called the tools it was supposed to and returned valid output. A second model acting as a judge scores the report against a separate rubric for each quality dimension, with written anchors for what a low or high score means, and returns a short explanation with each score. Faithfulness checks ask whether each claim follows from the data the agent retrieved, which is where confident but unsupported statements show up. Finally, the judge compares the new report side by side with a saved, previously accepted one to decide which is stronger. LangChain's LangSmith, the evaluation tool the post is promoting, links each score to its explanation and to a record of every step the agent took.

The most useful part of the post is a failure. After a small prompt change, the overall score dropped, and the team spent most of a week reverting and retrying even though the reports looked fine to them. When someone finally read the per-criterion comments, the cause was a conflict inside the rubric: the new reports cited more sources, which raised one score, but the sources were vague, which lowered another, and the average hid the trade. The judge's comment on one report already said that nine references were mostly unnamed while the score still gave it 0.7. After the rubric was rewritten to value named, verifiable sources over sheer count, the same report scored 0.3. Korni also describes a rubric that rewarded conciseness more clearly than completeness, which pushed reports to get shorter even when users wanted caveats and method.

Korni's conclusion is that a miscalibrated evaluation is worse than none because it gives false confidence, and that a score is a pointer to where to look rather than a verdict. The account is a single company's experience published by the vendor whose tool it uses, and it reports no figures on how often the checks catch real regressions. Its specific, checkable lesson is that the reasoning a judge model gives for each criterion can reveal a faulty rubric that an aggregate number conceals.