Default to serverless for AI agents, then let production data decide what moves
Source
datadoghq.com
Date
Why it matters
If agents idle most of their lifetime, always-on sandboxes waste money. Scale-to-zero compute is a cheaper first choice before moving heavy workloads to dedicated capacity.
Key takeaways · AI-distilled
Datadog cites a study of more than 35,000 production coding-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → sessions in which the median session ran LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → or tool calls for only 20% of its lifetime, sitting idle between steps with its sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → still allocated.
Datadog recommends comparing cost per successful run, not invocation price: 100 research-agent runs costing $12 with 80 completions is $0.15 per success, while dividing by all 100 starts gives $0.12 and hides failed and abandoned work.
For long runs, Datadog advises an external durable orchestrator to own workflow state, scheduling and resumption while the agent framework handles model and tool calls inside bounded steps, so a framework process can end without losing progress.
Daily averages mislead placement: 1,000 events a day looks like under one a minute, but if all arrive in a nightly batch a small always-on worker can be overwhelmed, so Datadog recommends profiling peak arrival rate, burst duration and idle time.
Datadog says to move only the constrained step: a conversational agent can stream its first reply from warm capacity and hand follow-up research to a serverless workflow, and warm workers do not fix latency from unnecessary retrieval or model turns.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
