Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain
- Source
- AI Engineer
- Author
- AI Engineer
- Date
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
It gives a decision rule for the prompt-tuning versus question and shows traces are the substrate that makes improvement measurable rather than anecdotal.
“Vivek Trivedy's argument is that observability and continual learning are the same problem in different clothing, because an agent acting in an environment produces the only real record of what happened, and that record is the substrate everything else is built on.”
“Working with Harvey on a legal benchmark, they found an open model could match their frontier model's trace judging at one to two orders of magnitude lower cost, arrived at through harness engineering that the traces themselves pointed to.”
“His rule for when to stop tuning prompts and start finetuning is speed of feedback: harness engineering answers in about two minutes, so you exhaust that ceiling first, finetune to break through it, then return to harness engineering.”
“He also argues that dense feedback is what agents lack most, since a benchmark returning only pass or fail gives an agent nothing to act on, while traces already hold the fine grained signal.”
videoAn AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases
videoBeating RL With Reflection: GEPA and Optimize Anything — Lakshya A. Agrawal, GEPA
videoHow We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases
videoAutoresearch Made Our Models 3x Faster — Tejas Bhakta, Morph
Checking sign-in…
Loading comments…