Vibeleaderboard
← All Intel
Intel / video

Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

Source
youtube.com
Author
AI Engineer
Date

Context

Braintrust's Doug Guthrie shows in an AI Engineer workshop how tracing, recording each step an takes, can feed a repair loop. The example is a Python support agent with 968 sample support traces that expose order lookup failures and broken escalation calls. Online automations grade incoming traces, and a feature called Topics groups them by task, sentiment and workflow issue; a custom facet targets failures the default issue classification misses.

The closing steps turn findings into changes. Braintrust's Loop assistant queries trace data with SQL, selected failures return to datasets so later changes can be checked against real cases, and a coding agent using the Braintrust command-line tool proposes code and scorer changes. An automation packages findings and evaluation evidence into a pull request for review. The workshop description presents this as a demonstration and gives no measure of improvement; only the description was reviewed.

Key takeaways · AI-distilled
  • The workshop's demo is a Python support agent built on the OpenAI Agents SDK. A batch of 968 sample support traces surfaces order lookup failures and broken escalation calls as the cases worth investigating.
  • Online automations grade incoming traces while Braintrust's Topics feature clusters patterns in tasks, sentiment and workflow issues. Guthrie adds a custom facet to catch failures that the default issue classification misses.
  • Braintrust's embedded Loop assistant queries trace data with SQL to examine representative failures, and selected examples go back into evaluation datasets so later changes are checked against real failure cases.
  • A closing demo exposes an agent evaluation to a playground, so collaborators can adjust prompts and models while execution stays on the evaluation server.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…