From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
This second Arize workshop by Laurie Voss, following the 101 session, shows a repair loop for an . Its demo app, a toy store, fails a request for toys under $6 because its search tool has no price filter. According to the description, Voss adds tracing with a coding agent, inspects the recorded steps to find the missing filter, has the coding agent add it, and then checks the new request in the trace view.
The larger argument is about scale: reading traces one by one stops working when an application produces millions of them. Evaluations compress behavior into scores, but people can still become the bottleneck. Voss introduces a tool called Signal that groups recurring problems and suggests fixes, such as GitHub issues, datasets, evaluators or proposed pull requests. Regression evaluations are the stated guard against a fix breaking behavior that already works. This account rests on the published description, which comes from the presenter's own side.
- Skills installed in the workshop repo let a coding agent both add Arize AX tracing to the Wonder Toys app and query the resulting trace data.
- The trace diagnosis looks past exceptions and HTTP errors to search quality problems, such as empty results and overly narrow filters.
- Laurie Voss argues reading traces one by one fails at millions of traces, and still leave people interpreting every failure. Arize's Signal adds a layer that groups recurring problems and suggests fixes.
- In the demo, Signal findings can become GitHub issues, evaluation datasets, new evaluators or proposed pull requests, and Signal itself is traced and has its own evaluation suite.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIAI Engineer
videoBuilding Self-Learning Loops for Your Agent — Fuad Ali, Arize AIAI Engineer
videoLet Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AIAI Engineer
Checking sign-in…
Loading comments…



