Provider bills key spend to API keys, not workflow steps. This shows how to attribute agent cost to traces and spot avoidable spend such as uncached prompt prefixes and full history re-sent every turn.
Key takeaways · AI-distilled
Datadog argues AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → spend depends on architecture as much as model choice: how many turns the loop runs, how many tools it calls, how much retrieved context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → it carries, and how much of that context is re-sent every turn.
Provider bills are keyed to API keys and models, not applications or workflow steps, so a higher bill cannot be traced to more traffic, a model change, poor cache use or an agentic loopThe cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.Full definition → without per-trace cost attribution.
Datadog's diagnostic rule: a bill up 40% on 40% more traffic is a different problem from one up 40% on flat traffic, which is why it pairs total spend with cost per request, cache hit ratio and cache reuse ratio.
The cache waste it flags usually comes from how requests are built: dynamic content that changes a reusable prompt prefix, inconsistent message formatting, re-created prompt input, and tool definitions that change between calls.
Its savings estimates state an explicit target, such as moving a cache hit ratio from 0% toward 55% or 80%, so engineers can judge whether the assumption fits their traffic before changing production code.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
