Your site, your rules: new AI traffic options for all customers
- Source
- blog.cloudflare.com
- Author
- Cloudflare
- Date

Context
Cloudflare's new bot controls separate three purposes: building a search index, performing a user's immediate task, and collecting material for model training. The announcement makes those choices available even on its Free tier, so publishers can set them separately rather than through one AI-bot setting.
A crawler can serve multiple purposes. Under the policy scheduled for September 15, 2026, the most restrictive applicable choice would govern its access, so blocking training could also block a crawler used for search. Publishers therefore need to consider the crawler's combined activities when weighing content protection against discoverability.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
“The deal between crawlers and website owners that had held up for 30 years — we crawl you, and you get referrals — was no longer true.”
“So you have to make a Faustian bargain: either show up in search and let AI train on you, or risk losing discoverability.”
“For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”
“You might trust Stripe, but you don't necessarily trust everyone who wired Stripe's tools into a weekend project.”
“Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.”
articleCloudflare open-sources Kitesurf, a Workers-based browser for agents
articleCloudflare opens private origins to WAF, caching, and Workers
articleCloudflare open-sources Void for deploying Vite apps and agent-built tools
articleCloudflare and OpenAI pilot: network signals improve AI search discovery
Checking sign-in…
Loading comments…


