Cloudflare's Bot Preference Sync addresses a mismatch between the bot policy a site owner configures and the preferences published in robots.txt, the file crawlers consult for access instructions. The announcement describes generating entries from category-level settings for search, agents and training, while preserving existing disallow directives.
Publishing a preference and blocking a request remain separate mechanisms. Sync aligns the published file with supported dashboard choices; it does not translate every custom security rule or make an uncooperative crawler obey the file. Site owners with individual exceptions or more complex rules may still need to maintain their own robots.txt.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.