The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
Marc Klingen, co-founder of Langfuse, an open-source tool for tracking and evaluating AI agents, says in an AI Engineer talk that improving an used to mean a person reading traces every week, and that agents can now do much of that loop. The loop has an online half, tracing and monitoring a live agent, and an offline half, datasets, experiments and , which are test sets and scoring rules that show whether a change helped.
According to the description, AI is already good at proposing fixes and teams are starting to let it maintain datasets and eval criteria too, while humans keep setting direction. In a demo, a coding agent uses Langfuse's agent skills to find that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a better version before it ships. It is a vendor demonstration of one case, and the summary reports no measurements of how well the approach works more broadly.
- Langfuse's Marc Klingen describes a reference agent stack that connects online tracing and monitoring with offline datasets, experiments and evals.
- In his view AI is already good at proposing fixes and teams are starting to let it maintain datasets and eval criteria, while humans keep setting direction.
- He recommends feeding implicit user signals, such as edits and approvals, back into the loop as feedback.
- In the demo, a coding agent using Langfuse agent skills finds that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a revised version before shipping it.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…






