Vibeleaderboard
← All Intel
Intel / blog

Say it once: introducing Bot Preference SynC

Source
blog.cloudflare.com
Author
Jin-Hee Lee
Date

Context

Cloudflare's Bot Preference Sync addresses a mismatch between the bot policy a site owner configures and the preferences published in robots.txt, the file crawlers consult for access instructions. The announcement describes generating entries from category-level settings for search, agents and training, while preserving existing disallow directives.

Publishing a preference and blocking a request remain separate mechanisms. Sync aligns the published file with supported dashboard choices; it does not translate every custom security rule or make an uncooperative crawler obey the file. Site owners with individual exceptions or more complex rules may still need to maintain their own robots.txt.

Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Read the source blog.cloudflare.com
Recommended reads
Comments

Checking sign-in…

Loading comments…