Building an evidence-grounded agentic security operations harness on Cloudflare
- Source
- blog.cloudflare.com
- Date

Context
Cloudflare says in its engineering post that its first security-triage prototype, one AI agent given a whole investigation, produced claims the evidence did not support. It treated a detection (a hypothesis) as proof of an attack, could drift to the wrong account or time range, and could not tell a lookup that timed out from one that found nothing.
The fix moves evidence gathering into fixed, versioned code that runs before any model. It collects a customer's detection history, traffic baseline and enforcement outcome, each stamped with source and time, into an evidence package. Specialist agents for traffic, customer history, global telemetry and threat intelligence must cite items from that package, and code checks each citation exists and supports the claim. A synthesis agent cannot fetch new evidence.
The advisory separates not checked, checked with no match, and checked with evidence of absence, and gives no classification when evidence is insufficient. Analysts remain responsible for decisions. This is Cloudflare's description of an early beta in Managed Defense, with no accuracy results reported.
- Cloudflare's first single-agent prototype hallucinated claims the evidence did not support. It saw three failure modes: detections treated as proof of an attack, scope drifting to the wrong account or time range, and lookup timeouts indistinguishable from "checked and not found."
- The harness runs no model in its front half: deterministic code runs fixed, versioned recon workflows and stores each item with source, version and timestamp, so a snapshot can be replayed and specialist disagreements reflect interpretation rather than retrieval.
- For deeper review a runs four parallel specialists (traffic, customer , global telemetry, threat intel). A synthesis agent merges their typed findings but cannot fetch new evidence or choose a classification outside an approved vocabulary.
- Specialists must cite items in a versioned evidence package, and application code checks that each citation exists, belongs to the investigation and supports the claim. Invalid findings are corrected or recorded as limitations.
- Advisories distinguish "not checked," "checked, no match" and "checked, evidence supports absence." When evidence is insufficient, the system makes no classification or disposition recommendation, and analysts stay responsible for the final decision.
- Application — Software arranged so that a person can use it; in this course, the software around a model that accepts your request, prepares the context and shows the reply.
- Harness engineering — Designing the full environment and feedback system around an agent.
- Coordinator — The person or agent who owns the seams of the map: writes assignments, receives and checks returns, resolves disagreements, and integrates the result or assigns the integration.
- Context and context window — Context is the information available for a generation; the context window is the capacity available to hold it.
Checking sign-in…
Loading comments…


