Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Prompt fixes can silently trade one metric for another. A trace-derived dataset plus CI-gated evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → catches regressions before merge.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoHow Bridgewater Turned 100 Million Documents Into an Edge - Sam Green & Aaron Linsky, Bridgewater
videoCan AI Understand a Distressed Credit? - Harneet Kaur, Axim Intelligence
videoProving Alpha: How a Hedge Fund Measures AI's ROI - Hari Kumar, Contour Asset Management
videoReal AI Agent Attacks and What Would Have Stopped Them — Andrey Rybka, Bloomberg
videoLet Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AIAI Engineer
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AIAI Engineer
videoBuilding Self-Learning Loops for Your Agent — Fuad Ali, Arize AIAI Engineer