Introducing LangSmith Tuned Evaluators They automatically score agent behavior i

LangSmith is LangChain's platform for recording and evaluating what AI agents do, and its new Tuned Evaluators are prebuilt graders that LangChain runs for the customer. A team attaches one to a tracing project, the store of recorded agent conversations, and LangChain's judge scores each eligible conversation and attaches the result and an explanation to it (LangChain blog). The first one, Perceived Error, looks for evidence that the agent made a mistake, misunderstood a request or took the conversation in the wrong direction. Some evidence is explicit, such as a user correcting the agent, repeating a request or rejecting an action. The rest is inferred from contradictory answers, acknowledged mistakes or problems left unresolved. LangChain's case for it is that most users never rate a conversation, so failures have to be read from the conversation itself.

The pitch is cost. Using a frontier model as a judge, a common way to grade another model's output, is accurate but expensive across every production conversation, so LangChain says teams have tended to grade only a sample. It says it post-trained a specialized model on labeled traces from conversational agents, and that in its benchmark this model beat every frontier model it tested while cutting evaluation cost by 82%, with savings reaching 98% in some early-partner workloads. The post does not name the models compared or publish the benchmark, so both figures are LangChain's own claims.

The details of how it runs set its role. A conversation becomes eligible after at least two exchanges between the user and the agent and a configured idle period, and the evaluation finishes within 12 hours after that, so it serves review after the fact rather than catching errors live. It is available on Plus and Cloud Enterprise plans in the US, with a charge for each successful evaluation and no charge for skipped or failed ones.