Intel / video

Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse

Source: youtube.com
Author: AI Engineer
Date: Published May 20, 2026

Context

An automated improvement process can remove useful behavior if its score does not reward that behavior. In a Langfuse experiment, the team asked an to improve a reusable procedure for moving prompts into its prompt-management system. Optimizing for fewer interaction steps led the agent to remove instructions to fetch current documentation.

Another omitted requirement concerned linking prompt versions to production traces, which helps teams connect changes with observed results. Because the did not reward that linkage, instructions encouraging it were also removed. The experiment, a small one reviewed by humans, shows why a shorter or higher-scoring workflow needs review against the actual requirements: optimization follows the specified target, including its omissions.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.

Key quotes

“skills are kind of like a formalized shortcut to make things more reliable where you historically would have built a workflow”
Marc Klingen

“looking at traces still like gets you to like 80% of the detail”
Marc Klingen

“the target function really matters. Like I think it's it sounds obvious, but for us defining like the right target function was very hard”
Marc Klingen

“I think it's not a surprise that I think nobody reads documentation themselves”
Marc Klingen

“Question is do you even need to ask? Like I think you only need to ask if the skill is kind of like more trusted than the public web. If it's like same trust level, then why even bother asking?”
Marc Klingen

Chapters

00:00Introduction to Marc Klingen and Langfuse
01:22Conceptual mental model for agent skills
04:04The problem: scaling documentation and onboarding for coding agents
09:03Six key learnings from building a Langfuse skill
09:12Learning 1: Looking at traces for debugging
10:47Learning 2: Helping agents navigate information (sitemaps/formats)
11:40Learning 3: Surfacing a search endpoint for docs
12:36Learning 4: Implementing basic evaluation (eval) setups
13:53Learning 5: Referencing rather than duplicating content
14:24Learning 6: Using auto-research loops with target functions
17:24Discussion on challenges: distribution, versioning, and target functions
19:21Roadmap and future outlook for agent automation
20:09Q&A session

Read the source www.youtube.com