Vibeleaderboard
← All Intel
Intel / video

From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

Source
youtube.com
Author
AI Engineer
Date

Context

This second Arize workshop by Laurie Voss, following the 101 session, shows a repair loop for an . Its demo app, a toy store, fails a request for toys under $6 because its search tool has no price filter. According to the description, Voss adds tracing with a coding agent, inspects the recorded steps to find the missing filter, has the coding agent add it, and then checks the new request in the trace view.

The larger argument is about scale: reading traces one by one stops working when an application produces millions of them. Evaluations compress behavior into scores, but people can still become the bottleneck. Voss introduces a tool called Signal that groups recurring problems and suggests fixes, such as GitHub issues, datasets, evaluators or proposed pull requests. Regression evaluations are the stated guard against a fix breaking behavior that already works. This account rests on the published description, which comes from the presenter's own side.

Key takeaways · AI-distilled
  • Skills installed in the workshop repo let a coding agent both add Arize AX tracing to the Wonder Toys app and query the resulting trace data.
  • The trace diagnosis looks past exceptions and HTTP errors to search quality problems, such as empty results and overly narrow filters.
  • Laurie Voss argues reading traces one by one fails at millions of traces, and still leave people interpreting every failure. Arize's Signal adds a layer that groups recurring problems and suggests fixes.
  • In the demo, Signal findings can become GitHub issues, evaluation datasets, new evaluators or proposed pull requests, and Signal itself is traced and has its own evaluation suite.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…