How we use Insights to improve the agent behind…

Context

Datadog, the monitoring company, sells Observability for teams running AI agents. Its Insights feature reads an agent's traces, the recorded sequence of model requests and tool calls in each run, to find recurring cost and reliability problems and recommend fixes. In this post, three Datadog engineers describe running Insights on the agent that powers Insights itself.

Two findings concerned wasted . A search tool sometimes returned far more content than the model needed, and because those results stayed in the conversation, the agent sent them to the model again in later requests. Five searches across two runs put roughly 80,000 tokens into model requests, which Datadog says describes the requests it inspected, not savings it achieved. The recommended fix was compact previews that keep the identifiers the agent needs to fetch full content when it wants more. The second finding involved , where a provider reuses work from processing the same opening of a request, as long as requests begin with identical content. Datadog's agent placed task-specific details before its unchanging instructions and tool definitions, so whenever those details changed the provider could not reuse its work and wrote nearly the whole prompt to the cache again. Putting the stable material first could save about $2,734 a month by Datadog's estimate, which it says it has not measured.

Insights also explained failing tool calls. The agent made the same invalid call to update an insight five times over about 18 minutes, each time leaving out a required object, until the run ended in error; retrying could not help because the missing information never changed. Across 193 sessions in the sample, the agent guessed or altered resource names it was supposed to copy exactly from its loader. The recommendation was for the tool to check each name against the loader's list and return the valid choices when it does not match, enforcing a rule already in the agent's instructions. For the preview fix, a Fix with Bits option handed the finding, the recommended change and the trace to a coding agent, which prepared a pull request with tests. Datadog says it validates such changes by replaying the affected investigations and checking that the agent still investigates correctly, not only that it uses fewer tokens.

Key takeaways · AI-distilled

Datadog found five broad search_llmobs_spans calls across two agent runs returned about 80,000 tokens that were then carried into model requests. A later insight recommended compact previews that keep identifiers, so the agent fetches full spans only when needed.
Task-specific details placed before tool definitions stopped providers from reusing the cached prompt prefix. Datadog recommends putting stable instructions and tool definitions first and projects about $2,734 a month in savings, which it says it has not measured.
One run made five identical invalid update_insight calls over about 18 minutes and ended in error after four retries: each call omitted the full insight object the tool required, so retrying unchanged could never succeed.
The agent guessed or altered resource names in 193 sampled sessions despite instructions to use exact names. Datadog's recommended fix moves the rule into the tool: check names against the skill loader's list and return the valid choices on a miss.
Datadog validates each fix by replaying the affected investigations and comparing tool output size, token use and answer quality, since fewer tokens only help if the agent still investigates correctly and completely.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
prompt caching — Reusing the model's processed form of a repeated prompt prefix so subsequent calls skip re-reading it, cutting cost and latency substantially.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.