Intel / article

Compare AI agent changes with Online Experiments in Agent Observability

Source: datadoghq.com
Date: Published Nov 4, 2026

Context

Datadog has added Online Experiments to its Observability product, a way to test a change to an AI agent on real users before rolling it out. A team keeps the current version of an agent as a control and sends a share of live traffic to a changed version, which might use a different prompt, model, retrieval method or tool. Assignment is random, and Datadog Feature Flags can adjust the split without redeploying the application. Scores from evaluators the team has set up in Agent Observability become experiment metrics alongside latency, cost, usage and errors, and Datadog computes the relative change in each with a confidence interval. Datadog says the feature is free for Agent Observability customers sending up to 40,000 spans a month.

The case for testing on live traffic is that offline tests only cover what a team thought to include. Building a good offline dataset means choosing representative requests and agreeing on what good behavior looks like, and a model acting as judge needs a rubric and a person checking its verdicts. Production traffic includes requests nobody anticipated. Random assignment matters because without it, a new version could look better simply because it happened to receive easier questions. The experiment also puts quality scores and operating measures in one comparison, which addresses a gap a Datadog advocate described in an earlier talk on monitoring AI apps: latency and error rates show an app is up, not whether its answers are right.

Datadog's example is its own Insights agent, which investigates agent behavior from trace evidence. The team split traffic evenly between OpenAI's GPT-5.6 Luna and the cheaper GPT-6 Luna. An LLM judge scored each investigation from 1 to 5 for accuracy, completeness and whether conclusions followed from the evidence. GPT-6 Luna's estimated cost was 44.3% lower, with a confidence interval, the range of differences consistent with the data, of 37.5% to 51.2% lower. Judged quality was 3.84 for the old model and 3.86 for the new one, and Datadog notes that close averages do not by themselves establish equal quality. Duration was an estimated 12.8% higher and workflow errors 13.6% higher, but those intervals ran from 4% lower to 30% higher and from 16% lower to 43% higher. That range covers both little difference and a meaningful increase, so those two results were inconclusive.

Before deciding whether to proceed, the team read individual traces. In the traces they reviewed, GPT-6 Luna supplied invalid tool parameters more often and usually recovered after retries, which the overall quality score did not reveal. Datadog says it addressed the tool-parameter problem before moving to the new model, and suggests turning the failed calls into offline regression tests. All of these figures come from Datadog's account of its own product on its own agent. The post does not give the number of investigations in the experiment, cost and token figures are estimates, and it does not report results after the fix.

Key takeaways · AI-distilled

Datadog ran the feature on itself, splitting traffic evenly between GPT-5.6 Luna and GPT-6 Luna in its Insights agent, with an LLM judge scoring investigations 1 to 5 on accuracy, completeness and whether conclusions followed from the evidence.
The cost result was decisive and the risk results were not: estimated cost fell 44.3% (interval 37.5% to 51.2%), while duration came in 12.8% higher and workflow errors 13.6% higher with intervals running from lower to far higher.
Quality scores landed at 3.84 against 3.86, and Datadog cautions that close averages alone do not establish equivalent quality.
Traces exposed what the aggregate score hid: the newer model supplied invalid tool parameters more often and usually recovered after retries, so the quality judge absorbed the cost of failed calls without reporting their rate.
Online Experiments is free for Agent Observability customers submitting up to 40,000 LLM spans per month, and Datadog Feature Flags manages the traffic split so exposure can change without redeploying the application.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

Read the source www.datadoghq.com