How we investigated prompt cache misses with Datadog
Source
datadoghq.com
Date
Why it matters
Shows how tiny request differences and dropped cache settings silently defeat prompt caching, and how to find them by tracing per-request costs. Useful for anyone paying for repeated system prompts and tool definitions.
Key takeaways · AI-distilled
A changing value early in a prompt, such as a timestamp before otherwise identical instructions, breaks prefix matching and blocks reuse of what follows. Datadog proposed putting stable instructions first, request-specific details after, and a cache boundary where supported.
One Go service built its tool list by iterating a map, which does not guarantee order, so the same tools arrived in different sequences and changed the cached prefix. The patch sorts tool names, though later requests still had to confirm more input was reused.
An option named WithPromptCaching was enabled, yet the Anthropic requests showed no cache activity: it added a beta header but left out the request-body controls that tell the provider what to cache.
Datadog saw cache reads on Anthropic's native Messages API and none on its OpenAI-compatible endpoint, which Anthropic documents as not supporting prompt cachingReusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.Full definition →. Switching routes meant first checking that message roles, tool calls and streamingSending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.Full definition → still worked.
In Datadog's examples, requests with no cache hits had estimated totals of $0.74 to $0.86 each, against $0.02 to $0.14 with hits. Because some providers charge extra for cache writes, the team compares whole-task cost, latency and accuracy before and after a change.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
