Lays out how to turn user frustration and scope drift into evaluator scores on traces, then feed failures into offline experiments to prevent regressions.
Key takeaways · AI-distilled
Langfuse's workshop turns user frustration in production, such as all-caps messages or a user saying an answer did not work, into evaluation signals found in traces rather than anecdotes lost in chat history.
The demo adds three evaluators to a support AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → built on the OpenAI Agents SDK: an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge for user disagreement, a code check for all caps, and a scope check comparing the request with the system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition →. Explicit variable mappings target each at the relevant messages.
A coding agent then analyzes seeded traces through the Langfuse CLI and agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →, surfacing unnecessary tool calls, retries, latency and gaps in evaluation coverage. The loop continues by turning observed failures into datasets for offline experiments and regression checks.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
videoHow Secrets Leak Through AI Agents (and How to Stop It) — Venice AI
videoFrom Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
videoBuild a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
videoLet Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
