Cloudflare's new Bot Preference Sync keeps your robots.txt automatically aligned with whatever AI bot preferences you've already set in the dashboard — Search, Agent, and Training.
No more maintaining a static file. Set it once. We keep it in sync. 🔄
https://t.co/bo174jyzvJ
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Crawler policy is now split by purpose, and the hand maintained file was the part most likely to be wrong.
