Your site, your rules: new AI traffic options for all customers
- Source
- blog.cloudflare.com
- Author
- Cloudflare
- Date
If you build or operate AI agents, crawlers, or search bots, this changes what you can legally/technically access on the web going forward — understand the Search//Training split, the new robots.txt signals, and the September 2026 default-block deadline before your bot traffic gets silently throttled.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
“The deal between crawlers and website owners that had held up for 30 years — we crawl you, and you get referrals — was no longer true.”
“So you have to make a Faustian bargain: either show up in search and let AI train on you, or risk losing discoverability.”
“For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”
“You might trust Stripe, but you don't necessarily trust everyone who wired Stripe's tools into a weekend project.”
“Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.”
postCloudflare AI Search goes GA with image embeddings and PDF OCR
articleWe are introducing Clef and Clef-flash, open-source decision models hosted on…
articleToday, we’re making the Cloudflare Monetization Gateway available as part of a…
articleCloudflare Auto Router picks cheaper models per request in AI Gateway
Checking sign-in…
Loading comments…