Cache hit rates explain a 9x provider cost gap for agentic tasks
- Source
- x.com
- Date
Cache hit rates and Cost per Task by provider are now live on Artificial Analysis: Cost per Task for GLM-5.3 (max) varies ~9x across providers, driven by a combination of cache hit rate and token pricing Agentic workloads resend the full conversation history every turn, so most input tokens are cacheable, and cache pricing and cache hit rates drive most of the gap in Cost per Task. We already use measured cache hit rates to calculate Cost per Task, and are now publishing them for each model endpoint we track. What's new: ➤ Cache hit rates are now visible for all first-party and third-party endpoints, measured by running multi-turn agentic coding tasks and reading cached tokens from each provider's usage response. Cache hit rate is the share of cacheable input tokens served from cache. ➤ Cost per Task is now available by provider for open weights models, calculated from each provider's input, output, cache hit and cache write prices, applied at its measured cache hit rate. To standardize across providers for a given model, we use the same input, output and cacheable token counts from the model's Intelligence Index run. Key takeaways: ➤ Cost per Task on GLM-5.3 (max) spans ~9x…
Context
A high prompt-cache hit rate does not necessarily mean a low AI bill. In Artificial Analysis's comparison, two providers serving GLM-5.3 handled at least 98% of cacheable input from cache, yet their calculated costs per task were $0.92 and $8.68. One discounted cached input; the other charged its full input price. The October 13 post explains that the comparison applies provider prices and measured cache hit rates to standardized counts. For agents that repeatedly send conversation history, both the amount reused and the price charged for that reuse determine the cost.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Checking sign-in…
Loading comments…