Intel / video

The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO

Source: youtube.com
Author: 20VC with Harry Stebbings
Date: Published Oct 17, 2026

Context

means sending each request to whichever model suits it, for example a cheap model for a greeting and a frontier model for hard reasoning. In this 20VC interview, Harrison Chase, co-founder and CEO of LangChain, argues that routing is not a commodity and that the useful question is where it sits: in a gateway, the single API that stands between an application and many models, or in the , the code that runs the model in a loop and calls tools.

The gateway's advantage, Chase says, is portability. Because routing there is just an API call, it can be slipped under a harness you cannot change, such as Claude Code. He prefers the harness because it knows more about the task: it can keep the prompt cache warm and rewrite the prompt for a different model's preferred style, which a standard chat-completions request does not carry. He also suggests, framing it as his guess, that most use of gateways such as OpenRouter is simply one API and key for models customers have already picked, and that smart routing is done far less than people think, usually inside harnesses built by companies like Factory. His split is that a domain-specific agent needs routing in the harness, while general chat or coding might be served by a router at the gateway.

His reason routing stays hard is that models change quickly: when a new frontier model arrives that is better at everything, the payoff from choosing among models drops. These are a LangChain founder's views, not measurements. A gateway company has published compatible caveats: OpenRouter's router benchmark post lists reasons routing can end up performing worse than a single model, including the cost of rebuilding the input cache after a switch and too little information in the prompt to judge difficulty. Chase's skepticism is about routing, not the gateway business itself; near the end he says LangChain decided against building a gateway about eighteen months ago because it saw little to add over OpenRouter, a choice he now calls wrong given how large demand has turned out to be.

Terms in this piece · Glossary

model routing — Sending each request to a model chosen by the difficulty of the task, rather than using one model for everything.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.

Read the source www.youtube.com