How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Gives a step-by-step evals workflow for agents: trace, write code and judge evals, test the judge, prove prompt fixes, gate in CI. It replaces 'looks good to me' with measurable checks.

• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

• videoEvals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse
• videoAn AI Agent Got Its User Banned From Resy — Keycard, JPMorgan Chase & AAuth
• videoTeach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
• videoInsurance Claims Are AI's Messiest Data Problem — Denys Linkov, Wisedocs

• videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIAI Engineer
• videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AIAI Engineer
• videoLet Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AIAI Engineer

Checking sign-in…

Loading comments…