Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
Braintrust's Doug Guthrie shows in an AI Engineer workshop how tracing, recording each step an takes, can feed a repair loop. The example is a Python support agent with 968 sample support traces that expose order lookup failures and broken escalation calls. Online automations grade incoming traces, and a feature called Topics groups them by task, sentiment and workflow issue; a custom facet targets failures the default issue classification misses.
The closing steps turn findings into changes. Braintrust's Loop assistant queries trace data with SQL, selected failures return to datasets so later changes can be checked against real cases, and a coding agent using the Braintrust command-line tool proposes code and scorer changes. An automation packages findings and evaluation evidence into a pull request for review. The workshop description presents this as a demonstration and gives no measure of improvement; only the description was reviewed.
- The workshop's demo is a Python support agent built on the OpenAI Agents SDK. A batch of 968 sample support traces surfaces order lookup failures and broken escalation calls as the cases worth investigating.
- Online automations grade incoming traces while Braintrust's Topics feature clusters patterns in tasks, sentiment and workflow issues. Guthrie adds a custom facet to catch failures that the default issue classification misses.
- Braintrust's embedded Loop assistant queries trace data with SQL to examine representative failures, and selected examples go back into evaluation datasets so later changes are checked against real failure cases.
- A closing demo exposes an agent evaluation to a playground, so collaborators can adjust prompts and models while execution stays on the evaluation server.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AIAI Engineer
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIAI Engineer
videoBuilding Self-Learning Loops for Your Agent — Fuad Ali, Arize AIAI Engineer
Checking sign-in…
Loading comments…



