Vibeleaderboard
← All Intel
Intel / article

How we investigated prompt cache misses with Datadog

Source
datadoghq.com
Date

Context

Datadog, which sells monitoring software, found that some of its own AI applications paid full price for repeated input even though appeared to be turned on. Prompt caching lets a model provider reuse the work it already did on the start of a request, called the prefix, when the next request begins with exactly the same content. In a post on its engineering blog, Datadog traced the misses to two kinds of fault: requests whose prefix changed slightly from call to call, and caching settings that never reached the provider. The post is a vendor writing about its own products, Cloud Cost Management and Agent Observability, but the faults it describes come from ordinary application code.

The prefix faults show how little it takes to break reuse. One chat service put the current time, organization details, feature flags and incident into its system instructions, so a timestamp that moved from 10:00 to 10:05 ahead of otherwise identical instructions could stop the provider reusing everything after it. Datadog's proposed fix puts stable instructions first and request-specific details after. The same service built its tool list from a Go map, which does not keep a fixed order, so the same tools could arrive as A, B, C in one request and B, A, C in the next. The patch sorts tool names, and Datadog notes that its test checks the code's behavior, not whether the provider then reused more input. The team also checked how often values changed before blaming them: a UTC date in one could explain a miss across a date change but not repeated misses through the day.

The settings faults were invisible from inside the application. An option called WithPromptCaching was enabled, yet the Anthropic requests Datadog inspected showed no cache activity, because the option added a beta header but left out the controls in the request body that tell the provider what to cache. Another application never passed its SDK's caching option to the component that built the provider request. Datadog's gateway could also reach Anthropic through a compatibility endpoint that accepts OpenAI-formatted requests, and Anthropic's documentation confirmed that route does not support prompt caching. Datadog saw cache reads on native Messages API traffic and none on the compatibility route, though it says that comparison did not establish how much any application would save by switching.

Datadog's examples give a sense of the stakes without measuring the overall result. Three requests with no cache hits each carried more than 180,000 input tokens, at estimated input costs of $0.74 to $0.82 apiece, while a cached sequence cost an estimated $0.02 to $0.14 per request in total against $0.74 to $0.86 without hits. Before trusting such figures, the team found that its records could mislead: the application and its gateway could both log the same call and double its cost, and a record missing cache fields could make caching look unused. The post says the fixes helped reduce costs but gives no aggregate saving, and it notes that provider bills, not estimated token costs, are needed to confirm one. It reads as Datadog applying the cache-hit and uncached-prefix checks its earlier Agent Observability cost post described to its own services.

Key takeaways · AI-distilled
  • A changing value early in a prompt, such as a timestamp before otherwise identical instructions, breaks prefix matching and blocks reuse of what follows. Datadog proposed putting stable instructions first, request-specific details after, and a cache boundary where supported.
  • One Go service built its tool list by iterating a map, which does not guarantee order, so the same tools arrived in different sequences and changed the cached prefix. The patch sorts tool names, though later requests still had to confirm more input was reused.
  • An option named WithPromptCaching was enabled, yet the Anthropic requests showed no cache activity: it added a beta header but left out the request-body controls that tell the provider what to cache.
  • Datadog saw cache reads on Anthropic's native Messages API and none on its OpenAI-compatible endpoint, which Anthropic documents as not supporting prompt caching. Switching routes meant first checking that message roles, tool calls and still worked.
  • In Datadog's examples, requests with no cache hits had estimated totals of $0.74 to $0.86 each, against $0.02 to $0.14 with hits. Because some providers charge extra for cache writes, the team compares whole-task cost, latency and accuracy before and after a change.
Terms in this piece · Glossary
  • prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
  • streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
Read the source www.datadoghq.com
Recommended reads
Comments

Checking sign-in…

Loading comments…