How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Shows a repeatable path from reading traces to a calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition →LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge with concrete pass/fail criteria, replacing vibe checks that fail to scale for agents.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
videoBuild Your Own Team of Managed Agents - Jack Wotherspoon, Google DeepMind
videoHow to Build an AI Code Review Agent - Daksh Gupta, Greptile
videoHow to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
videoCan Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIAI Engineer
videoHow to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AIAI Engineer
videoThe Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AIAI Engineer