Unify routed around OpenAI's 15 request per second cache ceiling

Prompt caching can make or break an agent product, according to the guest in a short LangChain clip about how Unify handles it. An agent calls a tool, then sends the tool's result back so the model can continue. If the earlier part of that conversation is still cached, the model reuses its previous work instead of processing everything again, which the guest says is 90% cheaper through the OpenAI API and also faster. Getting those cache hits, he says, basically decides whether a product works.

The limit he describes comes from how OpenAI finds the cache. By default it hashes the start of a request and routes on that, and a developer can also supply a prompt cache key that influences the routing. At high volume, he says, one key can only take so much traffic, which he puts at "I think" 15 requests per second. OpenAI's prompt caching guide, read in late October 2026, gives a different figure: cached states live on individual machines, traffic above 15 requests per minute can lead to overflow routing, and a request can reuse a cache only if it reaches a machine holding a matching entry. The guide also says that on GPT-5.6 and later OpenAI handles cache routing automatically.

LangChain's post says Unify built its own routing around the limit; in the clip the guest says they "did the math" and are at "like a 95% cache hit rate or something." The clip does not explain how that routing works, so the useful lesson is the problem rather than a recipe: an agent that sends heavy traffic through a single cache key can lose the discount it was counting on, and whether that still applies depends on the model in use.