Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

Records of an agent's actions can reveal failures that a final pass-or-fail score leaves unexplained. Vivek Trivedy of LangChain proposes using agents to examine other agents' traces, the records of their task execution. His talk describes investigating questions such as whether behavior deteriorates after conversation compaction and where users become dissatisfied.

The proposed improvement loop starts with actual task histories, uses them to identify problems, and adjusts the surrounding agent system before turning to model fine-tuning. A trace supplies intermediate evidence about how a result arose, making it more useful for diagnosis than the outcome alone.

“Vivek Trivedy's argument is that observability and continual learning are the same problem in different clothing, because an agent acting in an environment produces the only real record of what happened, and that record is the substrate everything else is built on.”

“Working with Harvey on a legal benchmark, they found an open model could match their frontier model's trace judging at one to two orders of magnitude lower cost, arrived at through harness engineering that the traces themselves pointed to.”

“His rule for when to stop tuning prompts and start finetuning is speed of feedback: harness engineering answers in about two minutes, so you exhaust that ceiling first, finetune to break through it, then return to harness engineering.”

“He also argues that dense feedback is what agents lack most, since a benchmark returning only pass or fail gives an agent nothing to act on, while traces already hold the fine grained signal.”