LangSmith Engine adds self-hosting, Slack and Linear, and a cheaper mode

LangSmith Engine is an agent that LangChain runs inside its LangSmith observability platform to do the tedious part of maintaining agents: reading production traces, the step-by-step records of each agent run, to find what went wrong. According to the release post, it groups related failures into issues, each with example runs, a timeline and a root cause, then proposes a prompt or code change as a pull request for review. It also packages the failing runs as dataset examples so a fix can be tested offline, and keeps monitoring for that issue to catch regressions.

The headline numbers are LangChain's own. It says Engine now scores over twice as well as at its May launch on IssueBench, an internal benchmark for finding and grouping issues, and that its fixes score 25% better on public benchmarks such as Terminal-Bench. The post does not publish IssueBench, so the first figure cannot be checked from outside.

The new features are mostly about fitting into existing workflows and budgets. Engine can post new issues to Slack, open tickets in Linear, close issues that have stopped appearing in traces, and point to the exact place in a trace where a problem occurred. A Reduced Analysis mode scans fewer traces to lower cost. Self-hosted LangSmith customers can now use Engine too, but only its orchestration runs inside the customer's private cloud; analysis still goes to LangSmith Intelligence, which LangChain describes as its managed, zero data retention inference service. Engine is available on Plus and Enterprise plans.

At the time, Engine did not test its own fixes: the post named automated verification against customer datasets as the next step. LangChain's October roundup (coverage) later described an Engine v2 that automatically tests and validates its proposed fixes.