← All IntelClip / AI AgentsThe wrong order: implement from memory, fail verification, then read the docs
From Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse · ≈5:48
Diagnoses three compounding costs of an unskilled agent — stale instrumentation, no discovery of which eval strategy fits the app, and a slow wrong-then-fix loop.
What’s in it
- Diagnoses three compounding costs of an unskilled agent — stale instrumentation, no discovery of which eval strategy fits the app, and a slow wrong-then-fix loop.
Clip transcript
this. So, I'll use an example uh when you just ask Cloud Code to add a language to a project, it just worked, but it was not working in the best way possible. So, for example, user ask add tracing to my agent and then Cloud Code kind of like implements the instrumentation based on the outdated pre-training context, then tries to verify whether tracing works, then realize, "Oh, it doesn't work." And then only in a second step fetches up-to-date information to then correct the issue. At the same time, how you add tracing or evaluation to a project, they can evaluate in like like gazillion different ways. Like online evals, offline evals, human in the loop. There are so many different things. And often the question is like, "What even is relevant for your application?" So, human agent kind of need to figure it out on the way, but agent is not tasked to help you like figure out what's the best thing for your for your application. So, main problems are outdated training data, the non-optimal setup because the agent wasn't really like primed to help you discover what to do for your for your app. And it's very slow because you first add instrumentation the wrong way, then you figure out it's wrong, and then you need to fetch more documentation to to fix the the issues.
Comments
Sign in to comment.
Loading comments…