Vibeleaderboard
← All Intel
Intel / blog

Your site, your rules: new AI traffic options for all customers

Source
blog.cloudflare.com
Author
Cloudflare
Date

Context

Cloudflare's new bot controls separate three purposes: building a search index, performing a user's immediate task, and collecting material for model training. The announcement makes those choices available even on its Free tier, so publishers can set them separately rather than through one AI-bot setting.

A crawler can serve multiple purposes. Under the policy scheduled for September 15, 2026, the most restrictive applicable choice would govern its access, so blocking training could also block a crawler used for search. Publishers therefore need to consider the crawler's combined activities when weighing content protection against discoverability.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Key quotes

“The deal between crawlers and website owners that had held up for 30 years — we crawl you, and you get referrals — was no longer true.”

“So you have to make a Faustian bargain: either show up in search and let AI train on you, or risk losing discoverability.”

“For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”

“You might trust Stripe, but you don't necessarily trust everyone who wired Stripe's tools into a weekend project.”

“Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.”

Read the source blog.cloudflare.com
More from Cloudflare
Recommended reads
Comments

Checking sign-in…

Loading comments…