Vibeleaderboard
← All Intel
Intel / video

From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

Source
youtube.com
Author
AI Engineer
Date

Context

A judge can fail an 's work for reasons unrelated to the work. In this AI Engineer workshop, Laurie Voss of Arize AI builds a financial-analysis agent and uses a correctness judge, a model that grades each report, which rejects all 13 of its reports. According to the video description, the agent had used live web research while the judge graded only from its own knowledge, without seeing what the agent had gathered.

When the collected sources are given to a faithfulness check, the description says the result becomes more informative: six reports faithful and seven not. The workshop then covers reading traces (step-by-step records of what the agent did), a cheap deterministic ticker check, narrow evaluators that each test one dimension, and comparing the judge against human labels. The description's takeaway is to read real traces and define success before automating grading. This account is based on the published description and chapter list. The 201 session continues the series.

Key takeaways · AI-distilled
  • Supplying the agent's collected research sources to a faithfulness evaluator turned a judge that rejected all 13 financial reports into a usable split of six faithful and seven unfaithful reports.
  • Reading OpenInference traces of the Claude Agent SDK agent exposed failed file writes, missing report content and excessive web searches.
  • Laurie Voss layers : a cheap deterministic ticker check first, then evaluators that each check one dimension with explicit criteria, tagged inputs, examples and binary labels, so a failure names what to fix.
  • Judges are compared with human annotations using precision and recall and checked for biases. Failing traces then become datasets, and controlled experiments compare the changed agent on the same cases.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…