Vibeleaderboard
← All Intel
Intel / article

Cloudflare syncs robots.txt with AI bot policy settings

Source
blog.cloudflare.com
Author
Cloudflare
Date

Context

Cloudflare's Bot Preference Sync makes a site's robots.txt say the same thing as the AI bot settings Cloudflare enforces for it. robots.txt is the public where a site states which crawlers may use it; cooperating crawlers honor it, but it blocks nothing by itself. Cloudflare can also block crawlers at its network edge, and since July 1, 2026 its dashboard has let site owners set separate policies for Search, and traffic. In the announcement, Cloudflare says those two layers can disagree, for example when robots.txt disallows a crawler that the enforcement rules let through, and that some crawlers treat the disagreement as grounds to ignore the stated preference or work around the block.

With the sync on, Cloudflare generates robots.txt directives from those category settings and keeps them current. If a site already has a robots.txt, the generated section is placed before the existing contents, so the owner's own Disallow lines stay in place. The crawlers named in each category come from bots Cloudflare tracks in its BotBase and are updated periodically. The sync works only at the category level and does not read custom rules, so an owner who grants one company a special exception can turn the sync off and edit the file to match.

The Training setting carries a policy change. Choosing Disallow writes a no-training preference into robots.txt. Crawlers that combine search and training under one identity keep access for search indexing only if they meet Cloudflare's transparency requirements: honoring the no-training preference, letting site owners opt out of AI summaries, giving URL-level visibility into which pages were made available for training along with search metrics, and showing publicly that disallowing training does not hurt traditional search results. Mixed-use crawlers that do not meet those terms are still blocked when training is disallowed, and Cloudflare says it tracks which operators meet the criteria in Cloudflare Radar.

Defaults decide how many sites this reaches. Cloudflare says the sync is on by default for new customers, and existing customers of its older managed robots.txt feature will be asked to confirm their preferences before switching. New sites that select "I monetize from pages with ads on this domain" get Training set to Disallow, which Cloudflare says keeps them in search while keeping their content out of training; other new sites start with no blocks or disallows. The announcement, shared on August 21, said the feature would reach every plan within the following week. A note added on September 15 points to a follow-up post about a new Accountable designation for Apple, Google and Microsoft and about how the Block and Disallow Training settings now work together. That follow-up was not part of this evidence, so some of these details may have changed since.

Terms in this piece · Glossary
  • File — A named unit of stored data that keeps its contents while no program is using it.
  • Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
Read the source blog.cloudflare.com
More from Cloudflare
Recommended reads
Comments

Checking sign-in…

Loading comments…