Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Shows how to feed traces and datasets to a coding agent, turn a failure into an evaluator, and replay failures against a dev endpoint before shipping a fix. This makes observability actionable instead of a dashboard.
Key takeaways · AI-distilled
Ali's Wonder Toys shopping assistant, built on the OpenAI Agents SDK, confidently returned products over the user's budget because of a broken price filter; that production trace is what a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → reads alongside the repository to explain the bug.
Arize Signal groups recurring failures, gathers evidence and can open investigations or pull requests, while Arize skills give a coding agent direct access to traces, datasets and evaluators.
Ali contrasts a fixed LLM judge rubric with a agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → that can inspect tool calls, code and outside context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. The budget bug becomes an evaluation that checks whether returned prices actually fall inside the requested range.
Terms in this piece · Glossary
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
videoAI Security Engineer Foundations + Certificate — Micah Silverman, Snyk
videoThe SDLC You Build Yourself — Andy Wong, Andrei Bocan & Shane Wolf, Atlassian
videoAgent Speedrun — Elizabeth Fuentes Leone & Sandhya Subramani, AWS
videoCan Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AIAI Engineer
videoFrom Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIAI Engineer
videoThe Self-Improving OSS Agent Stack — Marc Klingen, LangfuseAI Engineer