By crawlers' stated purpose, AI training rose from 22% of crawler requests Cloudflare saw in spring 2025 to 52% by June 2026, while heavily crawled categories such as Retail and Financial Services lost as much as 40% of human traffic in under a year.
Cloudflare says AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → traffic differs from training crawls: it follows human routines, with a weekly rhythm and a summer dip, and returns each time someone asks about the content, so it grows with questions asked rather than pages published.
Fewer than 1% of sites on Cloudflare block search crawlers while 17% block training, which led Cloudflare to split its AI-bot switch into Search, Agent and Training controls. Disallow AI Training, which Apple, Google and Microsoft committed to honor, keeps sites indexed.
Pay Per Use charges when content is used, not per crawl: each verified buyer defines use and price and reports every use, Cloudflare checks the reports, bills the buyer and pays the publisher, who sees what was used and, where reported, which questions surfaced it.
Monetization Gateway, a closed beta for eligible U.S. customers, lets sites price requests with Cloudflare Rules and answers with HTTP 402 via the open x402A payment protocol built on HTTP's long-unused 402 status code, letting a server charge per request so software agents can pay without an account.Full definition → protocol, charging agents per request, query or tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. Cloudflare's own AI Gateway uses it for inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
x402 — A payment protocol built on HTTP's long-unused 402 status code, letting a server charge per request so software agents can pay without an account.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Gives measured figures on how fast agent traffic is growing, which informs how services should be built, rate-limited and priced for agents.
