Clip / AI Agents

Lesson 1: reading traces gets you 80% of the way

From Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse · ≈9:02

They instrumented the coding agent itself and watched it use their product, then tuned the skill to stop it wandering — dogfooding the observability tool on the skill under development.

What’s in it

They instrumented the coding agent itself and watched it use their product, then tuned the skill to stop it wandering — dogfooding the observability tool on the skill under development.

Watch this moment

Clip transcript

So so just like there's there's a stark difference. And yeah, what did we learn on the way? Six main things. I'll go through every single one of them of what were basically our realizations when when building the skill. And like one, looking at traces still like gets you to like 80% of the detail. This was always what we kind of like tried to preach regarding like evals where many people try to complicate things right away while they haven't dig through through like just like what did the agent actually do at runtime themselves a couple of times. So so what did we do? We have instrumentation for cloud code and just ourselves interactively like tried to use LangFuse with cloud code and then look through traces in length use to understand whether there is an error, how can we improve the skill to make it like straight shooting at the goal instead of like wandering in different ways to the to the target. And that was really helpful, too.