Vibeleaderboard
← All Intel
Intel / video

The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse

Source
youtube.com
Author
AI Engineer
Date

Context

Marc Klingen, co-founder of Langfuse, an open-source tool for tracking and evaluating AI agents, says in an AI Engineer talk that improving an used to mean a person reading traces every week, and that agents can now do much of that loop. The loop has an online half, tracing and monitoring a live agent, and an offline half, datasets, experiments and , which are test sets and scoring rules that show whether a change helped.

According to the description, AI is already good at proposing fixes and teams are starting to let it maintain datasets and eval criteria too, while humans keep setting direction. In a demo, a coding agent uses Langfuse's agent skills to find that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a better version before it ships. It is a vendor demonstration of one case, and the summary reports no measurements of how well the approach works more broadly.

Key takeaways · AI-distilled
  • Langfuse's Marc Klingen describes a reference agent stack that connects online tracing and monitoring with offline datasets, experiments and evals.
  • In his view AI is already good at proposing fixes and teams are starting to let it maintain datasets and eval criteria, while humans keep setting direction.
  • He recommends feeding implicit user signals, such as edits and approvals, back into the loop as feedback.
  • In the demo, a coding agent using Langfuse agent skills finds that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a revised version before shipping it.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from AI Engineer
Recommended reads
Comments

Checking sign-in…

Loading comments…