Cloudflare Lets Sites Block AI Training While Staying Search-Indexed

Cloudflare shipped a Disallow AI Training toggle that site owners can turn on without losing search visibility. Until now, blocking AI crawlers from training on a site's content through robots.txt directives risked also blocking the same crawlers from indexing the site for search, since many crawlers serve both purposes under one user agent. The new toggle separates the two: a site can opt out of having its content used for model training while staying discoverable in search results. Cloudflare paired the feature with an Accountable designation for crawlers that respect the distinction, shared with Apple, Google, and Microsoft, which puts those four companies behind a shared standard for how a crawler should identify and behave. For publishers, the practical change is that opting out of AI training no longer means an implicit trade-off with search traffic, which removes the main reason many sites left crawling wide open despite objecting to training use.