Default to serverless for AI agents, then let production data decide what moves
- Source
- datadoghq.com
- Date
Context
Datadog argues that teams should start AI agents on serverless compute, which a cloud platform supplies as requests arrive and scales to zero when they stop, because many agents spend much of their time waiting. Its post cites a study by researchers at the Hong Kong University of Science and Technology, Alibaba and ByteDance that analyzed a day of production traces from one deployed coding , 35,037 sessions in all. The median session was running model or tool calls for only 20% of its lifetime. The rest was idle time between steps, for example waiting for an approval, a 's result or the next instruction, with the session's still reserved. An always-on server is paid for through every one of those waits.
The test Datadog proposes is the ratio of active compute time to total elapsed time for a run, estimated from prototypes or load tests at first and then from production measurements. A low ratio favors on-demand execution; a high ratio with steady traffic can make dedicated workers cheaper. For runs that span many steps or pause for a person, the post recommends a durable serverless workflow, using services such as AWS Step Functions or Azure Durable Functions, where an orchestrator records progress after each bounded step, releases compute during a wait and resumes from that checkpoint. A step that must keep a process alive, such as a reply or holding a persistent connection, cannot pause that way and may need capacity kept warm.
Datadog also says to compare options by cost per successful run rather than by invocation price. In its worked example, a research agent starts 100 runs that cost $12 in total, but only 80 finish, so each successful run costs $0.15 rather than the $0.12 an average over all runs would suggest. When latency or steady load does point away from serverless, the post advises moving only the affected step, for example keeping a warm service to stream a chat agent's first reply while the follow-up research runs as a serverless workflow. The post presents this as a decision method to test against a team's own workloads; it reports no measurements from Datadog's own agents, and it recommends Datadog's monitoring products for taking the measurements.
- Datadog cites a study of more than 35,000 production coding-agent sessions in which the median session ran LLM or tool calls for only 20% of its lifetime, sitting idle between steps with its sandbox still allocated.
- Datadog recommends comparing cost per successful run, not invocation price: 100 research-agent runs costing $12 with 80 completions is $0.15 per success, while dividing by all 100 starts gives $0.12 and hides failed and abandoned work.
- For long runs, Datadog advises an external durable orchestrator to own workflow state, scheduling and resumption while the agent framework handles model and tool calls inside bounded steps, so a framework process can end without losing progress.
- Daily averages mislead placement: 1,000 events a day looks like under one a minute, but if all arrive in a nightly batch a small always-on worker can be overwhelmed, so Datadog recommends profiling peak arrival rate, burst duration and idle time.
- Datadog says to move only the constrained step: a conversational agent can stream its first reply from warm capacity and hand follow-up research to a serverless workflow, and warm workers do not fix latency from unnecessary retrieval or model turns.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
- sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
- streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
Checking sign-in…
Loading comments…