LangSmith Fine-Tuning turns agent traces into fine-tuned models
- Source
- video.twimg.com
- Author
- LangChain
- Date

Context
LangSmith , now in public beta from LangChain, turns recorded runs of an doing a job well into training examples for a specialized model. The method is supervised fine-tuning, which teaches a model by showing it worked examples. According to the launch article, each recorded run, or trajectory, keeps the ordered messages, tool calls and tool results together with the tool definitions, so every action stays paired with what the model saw at that turn. The demo video explains why that matters: for long-running agents, the available tools can change from turn to turn.
The smithtune command-line tool runs the rest. In the demo it pulls candidate runs, has the user and a coding agent write a rubric for a good run, and asks two judging models that must both agree before a run is kept. It then splits the data for training and testing, trains with Baseten or Fireworks, and compares the tuned model with the original. The project's repository documentation says this comparison is a replay that predicts the next response or tool call from recorded without executing generated tool calls, so its scores measure agreement with recorded behavior, not whether the model completes real tasks.
The demo's own result was small. On five held-out test examples, the tuned model agreed slightly more often than the base model, and the presenter said a job that size should not be expected to show much benefit. The account here rests on an automatic speech transcript of the demo post video, without its visuals, and only the opening post of the thread was collected.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…





